Open data opens fascinating views into recent history and near present day in Europe.
This blog post investigates European Social Study 2002-2016 data and explores subjective happiness, trust in European Parliament and views on immigration and refugees by using statistical analysis and modelling. Separate data from World Bank and UNHCR is used as reference.
European Social Survey
Is an interview-based study that has been running every two years since 2002 in most European countries. A few countries have been in all surveys. The purpose of the study is described in the abstract of its last eighth round (conducted 2016-2017) as
The European Social Survey (ESS) is an academically-driven multi-country survey, which has been administered in over 30 countries to date. Its three aims are, firstly – to monitor and interpret changing public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions, secondly – to advance and consolidate improved methods of cross-national survey measurement in Europe and beyond, and thirdly – to develop a series of European social indicators, including attitudinal indicators.
Survey material and data is freely usable and was the basis for this exploration of Europe in the first few years of a new century.
According to ESS data there seems to be a borderline running through Europe dividing it diagonally into higher and lower subjective happiness domains
Even if there are a few countries with statistically significant changes in subjective happiness between first and last ESS survey (or between last and previous) the effect size most often is negligible as seen in the following table
|cntry name||2002||2014||2016||d0216||eff cil 0216||eff ciu 0216||eff m 0216||p 0216||sig 0216||d1416||eff cil 1416||eff ciu 1416||eff m 1416||p 1416||sig 1416|
- name of country
- 2002, 2014, 2016 – average subjective happiness in ESS that year
- d0216, d1416 – difference between two surveys, 2002 vs. 2016 and 2014 vs. 2016
- eff cil 0216, eff ciu 0216, eff cil 1416, eff ciu 1416 – confidence intervals for effect size with cil denotes lower and ciu upper end of 95% confidence interval
- eff m 0216, eff m 1416 – assessment on the meaning of the difference
- p 0216, p1416 – statistical significance of the difference, numeric p-value
- sig 0216, sig 1416 – statistical significance levels: + 0.1, * 0.05, ** 0.01, *** 0.001
Effect was estimated using Cohen’s d metric. Statistical significance with weighted t-test.
Factors associated with subjective happiness
There was a linkage between subjective happiness and trust in European Parliament in the first version of the blog, but not very clear. In the updated analysis, now using multiple linear regression model, trust in European Parliament did not end up as a factor by the algorithm used.
Most important factor (and positive one) was willingness of people to help others.
Second largest factor (but a negative one) was trust in United Nations. This provides a link to trust in European Parliament which turned out to be the most important and positive factor in the linear model on trust in United Nations.
There seems to be linkage between subjective happiness and these international institutions.
Other key factors identified (in the end of the blog post there is a tabulation of the factors along with explanation of abbreviations used)
- stfhlth – State of health services in country nowadays (+)
- stfedu – State of education in country nowadays (+)
- eisced2 – Level of education (-)
- stfeco – Satisfaction with present state of economy in country (+)
Education level having a negative impact is curious and would need further investigation to understand it better.
The model was trained with data from 2002 to 2014 and tested against 2016 results. It performed decently even though it underestimates the real score in Poland and is almost missing that of Spain.
Then again, other types of models were not tried since a linear model often provides fairly easy way to get insight into associations.
In other models created for this reboot of the blog post, I decided that subjective happiness is not a plausible explanatory factor. It is more naturally as an ultimate outcome and depends on other factors. Therefore, it did not show up in any models as independent variable. But somebody else might have decided otherwise.
Trust in European Parliament
Most important – and positive – factors in this model were trust in United Nations, satisfaction with government and increasing of integration of European Union.
Interestingly, the most important negative factors were willingness of people to help others, question on whether immigrants are good for economy and satisfaction with economy.
The model was tested against 2016 results and it performed again relatively well – even though the Portuguese were not giving European Parliament as much trust as the model estimated.
Overall average from ESS surveys gives an impression that trust in European Parliament is highest in the Nordic countries and a few smaller countries. But on the scale of zero to 10 the scores are modest everywhere and there are differences. Some easier and other not so obvious to understand.
Recent history can be seen in the timeline results below easily for Greece which has had its economic crisis and challenges with aid from European Union. Low scores from Russia could be linked to the crisis in Crimea. Overall, the graph shows intriguing trajectories, each with a story of its own to tell.
Sample cases: Greece and Ukraine
Greece has not been in the survey since 2010. That was the year when the first economic support package from EU was agreed. As one can see, most indicators were sliding down towards 2010. And it is worth noting that initially trust in European Parliament was quite high – but dropped low. Trust in legal system, trust in police and – no surprise – trust in country’s economy went down as well.
Recent history has been quite difficult for Ukraine starting with various political crisis to still ongoing dispute on Crimea (which started 2014, i.e. after the last survey Ukraine participated in).
Happiness score increased while trust in European Parliament decreased, even though not much. However, trust in police and legal system seems to have taken big hit -and there was very low level of interest in politics reported in 2008.
A division of Europe
A k-means clustering exercise was done with a selection of indicators in ESS and it chose to divide Europe into two groups: one consisting of Nordic countries together with Belgium, The Netherlands, Ireland, Switzerland and Austria. And the other with all the rest.
The selected indicators show quite clear differentiation between the groups. Here we have converted absolute numerical ESS scores into uniform range to make it easier to compare them on one graph.
In most of the indicators countries in cluster 2 give higher scores which roughly translate to “more” or “better” — with a couple of noteworthy exceptions
- health – low value is good and high value is poor health
- impcntrl – low value means more immigration and high less immigration from poor countries
No formal clustering exercise was done with World Bank Development indices or UNHCR data which were used for context
However, visual box plot comparisons of selected indicators seem to support the clustering.
- Cluster 2 countries appear more urbanized than cluster 1
- Proportion of population born in another country is higher in cluster 2 even though there is quite some overlap in distributions
- There is higher percentage of people active in workforce in cluster 2
- Government health and education expenditures vs. GDP are higher in cluster 2
- GDP growth (and per capita growth) is higher in cluster 1 countries but there is significant overlap with cluster 2 distribution
- significantly higher share of population is using Internet in cluster 2 compared to cluster 1
- tax revenues as % of GDP are higher in cluster 2
- Personal remittances received (% of GDP) is higher in cluster 1
Since there are big differences between countries and the indicators, a logarithmic scale was chosen. And to make that work all scores were shifted by one. Real scores are one percentage unit lower than shown below.
Here we can see clear difference regarding percentage of refugees and asylum seekers in population, being significantly higher in the countries of cluster 2.
Migration and refugees
Refugees, asylum seekers and immigration has been a hot topic in recent history – and still is.
There are quite different views on the topic of immigration from poor countries outside of Europe as seen in the timeline below. On one end of the spectrum is Hungary (cluster 1) with strongly restrictive views and on the other Sweden (cluster 2) with more open approach.
The views on benefits of immigration to country’s economy vary a lot as well. However, this is not necessarily the same as views on immigration from poor countries, at least not for all the countries covered in ESS.
But for some it could since the countries with restrictive views on immigration like Hungary and Cyprus are getting low scores while Sweden is on the high end.
UNHCR data gives some perspective into the recent history on refugees in Europe showing quite significant increases in some countries – and in some others not that much when compared to whole population of each country.
Here it looks like Sweden is acting consistently with the responses given in ESS.
There are some curious jumps in e.g. Germany which I suspect coming from changes in classification but have not had chance to verify.
GDP growth per capita is another interesting backdrop to recent history of Europe. The worldwide economic crisis around 2008 is clearly visible in practically all countries and could impact ESS responses around that time frame, even though I’ve not made any formal analysis on that – perhaps later.
The huge spike in 2015 for Ireland was caused by change in legislation causing massive one-off reporting by big companies based in Ireland.
ESS data is an interesting source for getting insights into the recent history and ongoing present of Europe from data.
In this post perhaps the most interesting – but perhaps not surprising – findings were
- Europe can be split into two groups of countries, apparently with different views on life in general and on immigration and refugees
- major explaining factors for subjective happiness were willingness of people to help other (positive, not surprising) and trust in United Nations (negative, somewhat strange, indirectly linked to trust in European Parliament as it is the main positive factor in trust in United Nations)
- major explaining factors for trust in European Parliament were trust in United Nations (positive), satisfaction in government (positive), further integration of European Union (positive), which all were perhaps not surprising
- views on immigration and refugees are polarized and it appears that those countries where immigration is seen positively impacting economy tend to be those which are more open to immigration from poor countries outside of Europe
Linking more between ESS and other data may give further insights and I will probably return to this topic later.
Data and scripts used here will eventually find its way into GitHub.
PS: Indicators and countries can be explored in an interactive application here
|happy||Taking all things together, how happy would you say you are||0||10||ESS||Extremely unhappy||Extremely happy|
|ppltrst||Most people can be trusted or you can’t be too careful||0||10||ESS||You can’t be too careful||Most people can be trusted|
|pplfair||Most people try to take advantage of you, or try to be fair||0||10||ESS||Most people try to take advantage of me||Most people try to be fair|
|pplhlp||Most of the time people helpful or mostly looking out for themselves||0||10||ESS||People mostly look out for themselves||People mostly try to be helpful|
|polintr||How interested in politics||1||4||ESS||Very interested||Not at all interested|
|trstprl||Trust in country’s parliament||0||10||ESS||No trust at all||Complete trust|
|trstlgl||Trust in the legal system||0||10||ESS||No trust at all||Complete trust|
|trstplc||Trust in the police||0||10||ESS||No trust at all||Complete trust|
|trstplt||Trust in politicians||0||10||ESS||No trust at all||Complete trust|
|trstprt||Trust in political parties||0||10||ESS||No trust at all||Complete trust|
|trstep||Trust in the European Parliament||0||10||ESS||No trust at all||Complete trust|
|trstun||Trust in the United Nations||0||10||ESS||No trust at all||Complete trust|
|stflife||How satisfied with life as a whole||0||10||ESS||Extremely dissatisfied||Extremely satisfied|
|stfeco||How satisfied with present state of economy in country||0||10||ESS||Extremely dissatisfied||Extremely satisfied|
|stfgov||How satisfied with the national government||0||10||ESS||Extremely dissatisfied||Extremely satisfied|
|stfdem||How satisfied with the way democracy works in country||0||10||ESS||Extremely dissatisfied||Extremely satisfied|
|stfedu||State of education in country nowadays||0||10||ESS||Extremely dissatisfied||Extremely satisfied|
|stfhlth||State of health services in country nowadays||0||10||ESS||Extremely dissatisfied||Extremely satisfied|
|eisced||Highest level of education, ES ISCED||1||7||ESS||ES-ISCED I , less than lower secondary||ES-ISCED V2, higher tertiary education, >= MA level|
|health||Subjective general health||1||5||ESS||Very good||Very bad|
|euftf||European Union: European unification go further or gone too far||0||10||ESS||Unification already gone too far||Unification go further|
|impcntr||Allow many/few immigrants from poorer countries outside Europe||1||4||ESS||Allow many to come and live here||Allow none|
|imbgeco||Immigration bad or good for country’s economy||0||10||ESS||Bad for the economy||Good for the economy|
|imueclt||Country’s cultural life undermined or enriched by immigrants||0||10||ESS||Cultural life undermined||Cultural life enriched|
|imwbcnt||Immigrants make country worse or better place to live||0||10||ESS||Worse place to live||Better place to live|
|BX.TRF.PWKR.DT.GD.ZS||Personal remittances, received (% of GDP)||0.070||8.154||WB||NA||NA|
|GC.TAX.TOTL.GD.ZS||Tax revenue (% of GDP)||1.230||48.344||WB||NA||NA|
|IT.NET.USER.ZS||Individuals using the Internet (% of population)||1.874||97.298||WB||NA||NA|
|NY.GDP.MKTP.KD.ZG||GDP growth (annual %)||-14.814||25.557||WB||NA||NA|
|NY.GDP.PCAP.KD.ZG||GDP per capita growth (annual %)||-14.560||24.765||WB||NA||NA|
|REFG.PCT||Refugees (% of population) – calculated||0.000||2.319||WB||NA||NA|
|SE.XPD.TOTL.GD.ZS||Government expenditure on education, total (% of GDP)||2.326||8.560||WB||NA||NA|
|SH.XPD.GHED.GD.ZS||Domestic general government health expenditure (% of GDP)||0.000||9.420||WB||NA||NA|
|SL.TLF.CACT.ZS||Labor force participation rate, total (% of total population ages 15+) (modeled ILO estimate)||47.853||68.565||WB||NA||NA|
|SM.POP.TOTL.ZS||International migrant stock (% of population)||0.795||29.387||WB||NA||NA|
|SP.RUR.TOTL.ZS||Rural population (% of total population)||2.103||50.373||WB||NA||NA|
|SP.URB.TOTL.IN.ZS||Urban population (% of total)||49.627||97.897||WB||NA||NA|
|UNHCR.Refugee.PCT||Refugees (incl. refugee-like situations)(% of population) – calculated||0.000||2.319||UNHCR||NA||NA|
|UNHCR.Asylum-.PCT||Asylum-seekers(% of population) – calculated||0.000||1.603||UNHCR||NA||NA|
|UNHCR.Interna.PCT||Internally displaced persons(% of population) – calculated||0.000||4.000||UNHCR||NA||NA|
|UNHCR.Statele.PCT||Stateless(% of population) – calculated||0.000||11.048||UNHCR||NA||NA|
|UNHCR.Other.PCT||Other population types together(% of population) – calculated||0.000||6.276||UNHCR||NA||NA|
A real serious study on the topic (I’ve yet to read myself):
- Europeans’ Personal and Social Wellbeing, Topline Results from Round 6 of the European Social Survey
- link: http://www.europeansocialsurvey.org/docs/findings/ESS6_toplines_issue_5_personal_and_social_wellbeing.pdf
Citation of data:
- European Social Survey Cumulative File, ESS 1-7 (2016). Data file edition 1.0. NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS REIC.
- European Social Survey Round 8 Data (2016). Data file edition 2.0. NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC
Citation of documentation:
- European Social Survey (2016). ESS 1-7, European Social Survey Cumulative File, Study Description. Bergen: NSD – Norwegian Centre for Research Data for ESS ERIC.
- European Social Survey (2016): ESS8- 2016 Documentation Report. Edition 2.0. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC
Distributor of Data
World Development Indicators https://data.worldbank.org/products/wdi
The primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates.
Excel file was downloaded and used
Population Statistics / Time Series http://popstats.unhcr.org/en/time_series
On this page, each row of data represents the information about UNHCR’s populations of concern for a given year and country of residence and/or origin. Data is presented as a yearly time series across the page. In the 2017 data, figures between 1 and 4 have been replaced with an asterisk (*). These represent situations where the figures are being kept confidential to protect the anonymity of individuals. Such figures are not included in any totals”
Csv extract of all data was used.
Updated 2018-06-20 – a few misspellings corrected and title of one graph corrected
Updated 2018-06-21 – fixed formatting issues in two tables
Updated 2018-08-14 – added Github link https://github.com/juhariis/ess to the R-markdown document for this blog entry exists, updated conclusions regarding the change of correlation (some less happy countries went down in their trust – reality bites?)
Updated 2018-09-02 – Sometimes there is a need to revisit work that one has done earlier. This blog post from June 2018 is one such item. When returning to this I realised that it very much needed an update. A much longer version – actually the R-studio workbook used for the investigation along with other material – is available in GitHub
Updated 2018-09-11 – added link to separate web application providing access to indicators by countries (and vice versa). The application is hosted at a free site with certain usage limits so it may stop working when the limits are reached. Sorry about that.
Updated 2018-09-13 – Replaced old maps with new versions, which are interactive in the copy of the workbook in Azure