{"id":21152,"date":"2024-01-29T15:06:09","date_gmt":"2024-01-29T15:06:09","guid":{"rendered":"https:\/\/www.economicsobservatory.com\/test\/?post_type=question&#038;p=21152"},"modified":"2024-01-29T17:10:17","modified_gmt":"2024-01-29T17:10:17","slug":"long-read-test","status":"publish","type":"question","link":"https:\/\/www.economicsobservatory.com\/test\/long-read-test","title":{"rendered":"Long Read &#8211; NHS England Data"},"content":{"rendered":"\n<p>[A&E Map]<\/p>\n\n\n\n<ul>\n<li>One of the NHS's key performance indicators is A&E Performance - 4 hour target<\/li>\n\n\n\n<li>The Data is available online but hard to access <\/li>\n\n\n\n<li>This explains how to find, clean and present NHS England waiting time data<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">NHS England Data<\/h2>\n\n\n\n<p>NHS England publishes its A&E attendance and admissions statistics, alongside other data, on its website, <a href=\"https:\/\/www.england.nhs.uk\/statistics\/statistical-work-areas\/ae-waiting-times-and-activity\/\">here<\/a>. On the second Thursday of each month, the previous month's performance statistics are published. These data report adherence to the 4 hour A&E target at the system and provider levels and break down admissions by department types. <\/p>\n\n\n\n<p>Unfortunately, each NHS release is just a cross-section; each month's data is released on its own, making comparisons over time difficult. To build a panel of NHS trusts' A&E performance, we must download and merge 13 years of separate monthly releases.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1. Finding A&E Stats Index Pages<\/h3>\n\n\n\n<p>The data is indexed with a separate page for each fiscal year. For example, '<a href=\"https:\/\/www.england.nhs.uk\/statistics\/statistical-work-areas\/ae-waiting-times-and-activity\/ae-attendances-and-emergency-admissions-2022-23\/\">A&E Attendances and Emergency Admissions 2022-23<\/a>'  hosts individual monthly A&E data releases for April 2022-March 2023.<\/p>\n\n\n\n<p>The first step in collecting and merging the data we need is to identify the URLs of the index pages where the monthly releases are hosted. In an ideal world, these URLs would be standardised but this is not the case. Instead we can collect the links by scraping them from the sidebar where they are listed.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"551\" src=\"https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-1024x551.png\" alt=\"\" class=\"wp-image-21154\" srcset=\"https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-1024x551.png 1024w, https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-300x161.png 300w, https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-768x413.png 768w, https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-1536x826.png 1536w, https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-2048x1102.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>We can extract the sidebar links we need by downloading the HTML source of the A&E stats homepage and then extracting all the links to the yearly index pages (e.g. \"<a href=\"https:\/\/www.england.nhs.uk\/statistics\/statistical-work-areas\/ae-waiting-times-and-activity\/ae-attendances-and-emergency-admissions-2023-24\/\">A&E Attendances and Emergency Admissions 2023-24<\/a>\"). In the code snippet below:<\/p>\n\n\n\n<script src=\"https:\/\/gist.github.com\/FM-ds\/8cc460dae7eb4e4c160c27b468ab60e1.js\"><\/script>\n\n\n\n<ol>\n<li>Lines 1-2 declare the location of the A&E Stats homepage and download the contents of the page, storing the request in req.<\/li>\n\n\n\n<li>Line 3 parses the page contents with <a href=\"https:\/\/pypi.org\/project\/beautifulsoup4\/\">BeautifulSoup<\/a>, creating a representation of the page that we can search through.<\/li>\n\n\n\n<li>Lines 4-5 find all the links on the page (by searching for instances of the HTML <a> tag which denote links) and filters to only those linking to \"A&E Attendances and Emergency Admissions\" pages.<\/li>\n<\/ol>\n\n\n\n<p>This yields us a list of every A&E stats index page, all the way from <a href=\"https:\/\/www.england.nhs.uk\/statistics\/statistical-work-areas\/ae-waiting-times-and-activity\/weekly-ae-sitreps-2010-11\/\">2011<\/a> to <a href=\"https:\/\/www.england.nhs.uk\/statistics\/statistical-work-areas\/ae-waiting-times-and-activity\/ae-attendances-and-emergency-admissions-2023-24\/\">2024<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. Batch Downloading Every Data Release<\/h3>\n\n\n\n<p>Now that we have a list of every A&E stats index page, we can automate the downloading of every Excel or CSV file they link to. Again, this is easily achievable in Python with BeautifulSoup and the built-in requests module.<\/p>\n\n\n\n<script src=\"https:\/\/gist.github.com\/FM-ds\/0967646d36bafcc40ab3b62815f165f1.js\"><\/script>\n\n\n\n<p>For every index page (found in step 1), we every link and filter for just the links to Excel and CSV files. We then download these, one by one. Following this, we have roughly 500 sheets of A&E statistics for the years 2010-2024.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Cleaning and Merging the Downloaded Data<\/h3>\n","protected":false},"featured_media":0,"template":"","categories":[],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Long Read - NHS England Data - Economics Observatory<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.economicsobservatory.com\/test\/long-read-test\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Long Read - NHS England Data - Economics Observatory\" \/>\n<meta property=\"og:description\" content=\"[A&amp;E Map] NHS England Data NHS England publishes its A&amp;E attendance and admissions statistics, alongside other data, on its website, here. On the second Thursday of each month, the previous month&#039;s performance statistics are published. These data report adherence to the 4 hour A&amp;E target at the system and provider levels and break down admissions [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.economicsobservatory.com\/test\/long-read-test\" \/>\n<meta property=\"og:site_name\" content=\"Economics Observatory\" \/>\n<meta property=\"article:modified_time\" content=\"2024-01-29T17:10:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-1024x551.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@econobservatory\" \/>\n<meta name=\"twitter:label1\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/long-read-test\",\"url\":\"https:\/\/www.economicsobservatory.com\/test\/long-read-test\",\"name\":\"Long Read - NHS England Data - Economics Observatory\",\"isPartOf\":{\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/#website\"},\"datePublished\":\"2024-01-29T15:06:09+00:00\",\"dateModified\":\"2024-01-29T17:10:17+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/long-read-test#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.economicsobservatory.com\/test\/long-read-test\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/long-read-test#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.economicsobservatory.com\/test\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Long Read &#8211; NHS England Data\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/#website\",\"url\":\"https:\/\/www.economicsobservatory.com\/test\/\",\"name\":\"Economics Observatory\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.economicsobservatory.com\/test\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/#organization\",\"name\":\"Economics Observatory\",\"url\":\"https:\/\/www.economicsobservatory.com\/test\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.economicsobservatory.com\/wp-content\/uploads\/2021\/06\/Logo-for-Twitter.png\",\"contentUrl\":\"https:\/\/www.economicsobservatory.com\/wp-content\/uploads\/2021\/06\/Logo-for-Twitter.png\",\"width\":540,\"height\":392,\"caption\":\"Economics Observatory\"},\"image\":{\"@id\":\"https:\/\/www.economicsobservatory.com\/test\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/twitter.com\/econobservatory\",\"https:\/\/www.linkedin.com\/company\/economics-observatory\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Long Read - NHS England Data - Economics Observatory","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.economicsobservatory.com\/test\/long-read-test","og_locale":"en_GB","og_type":"article","og_title":"Long Read - NHS England Data - Economics Observatory","og_description":"[A&E Map] NHS England Data NHS England publishes its A&E attendance and admissions statistics, alongside other data, on its website, here. On the second Thursday of each month, the previous month's performance statistics are published. These data report adherence to the 4 hour A&E target at the system and provider levels and break down admissions [&hellip;]","og_url":"https:\/\/www.economicsobservatory.com\/test\/long-read-test","og_site_name":"Economics Observatory","article_modified_time":"2024-01-29T17:10:17+00:00","og_image":[{"url":"https:\/\/www.economicsobservatory.com\/test\/wp-content\/uploads\/2024\/01\/image-1-1024x551.png"}],"twitter_card":"summary_large_image","twitter_site":"@econobservatory","twitter_misc":{"Estimated reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.economicsobservatory.com\/test\/long-read-test","url":"https:\/\/www.economicsobservatory.com\/test\/long-read-test","name":"Long Read - NHS England Data - Economics Observatory","isPartOf":{"@id":"https:\/\/www.economicsobservatory.com\/test\/#website"},"datePublished":"2024-01-29T15:06:09+00:00","dateModified":"2024-01-29T17:10:17+00:00","breadcrumb":{"@id":"https:\/\/www.economicsobservatory.com\/test\/long-read-test#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.economicsobservatory.com\/test\/long-read-test"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.economicsobservatory.com\/test\/long-read-test#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.economicsobservatory.com\/test\/"},{"@type":"ListItem","position":2,"name":"Long Read &#8211; NHS England Data"}]},{"@type":"WebSite","@id":"https:\/\/www.economicsobservatory.com\/test\/#website","url":"https:\/\/www.economicsobservatory.com\/test\/","name":"Economics Observatory","description":"","publisher":{"@id":"https:\/\/www.economicsobservatory.com\/test\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.economicsobservatory.com\/test\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/www.economicsobservatory.com\/test\/#organization","name":"Economics Observatory","url":"https:\/\/www.economicsobservatory.com\/test\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.economicsobservatory.com\/test\/#\/schema\/logo\/image\/","url":"https:\/\/www.economicsobservatory.com\/wp-content\/uploads\/2021\/06\/Logo-for-Twitter.png","contentUrl":"https:\/\/www.economicsobservatory.com\/wp-content\/uploads\/2021\/06\/Logo-for-Twitter.png","width":540,"height":392,"caption":"Economics Observatory"},"image":{"@id":"https:\/\/www.economicsobservatory.com\/test\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/twitter.com\/econobservatory","https:\/\/www.linkedin.com\/company\/economics-observatory\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/question\/21152"}],"collection":[{"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/question"}],"about":[{"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/types\/question"}],"version-history":[{"count":8,"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/question\/21152\/revisions"}],"predecessor-version":[{"id":21168,"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/question\/21152\/revisions\/21168"}],"wp:attachment":[{"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/media?parent=21152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/categories?post=21152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.economicsobservatory.com\/test\/wp-json\/wp\/v2\/tags?post=21152"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}