1. Abstract
New play-by-play data from the 2019 U Sports and CFL seasons was added to the Passes & Patterns Database. 125 games and 21,549 plays were added to the U Sports database, while 95 games and 14,025 plays were added to the CFL database. All data was cleaned and error-checked to fit with existing data format standards. A new scraper was developed in Python to streamline the CFL data collection process.
2. Introduction
With the end of each football season in November it becomes necessary to incorporate the newest season of data into the existing database (Clement 2018a, [c] 2018). Passes & Patterns maintains its database of every U Sports game it can find, dating back to 2002; this is the data on which the bulk of the original research is done. Passes & Patterns also keeps a parallel dataset of CFL data, on which similar analyses may be performed.
The addition of new data provides a validation set for existing models and allows these models to be refit for better accuracy. They also permit historiographical work to look at the development of the game over time
3. Data
The U Sports data was acquired from the usual sources, the conference websites (CWUAA n.d.; OUA n.d.; RSEQ n.d.; AUS n.d.). This is because the U Sports site itself strips the down, distance, and field position information from their data before posting the play-by-play data (“U SPORTS” n.d.). The CWUAA, OUA, and AUS all continue to use the Presto Sports format, known as data format 1 (PrestoSports n.d.), with minor variations between the conferences, while the RSEQ continues to use The Automated Scorebook, data format 3 (The Automated Scorebook n.d.). The data for each game was copied and pasted into Microsoft Excel, and the bulk of the data cleaning was accomplished with a number of VBA scripts, with an ocular review, finally the parse’s error-checking made final corrections. The author would like to note that the general quality of scorekeeping in U Sports has improved significantly over the past 20 years.
The new U Sports data incorporated 125 games; 108 regular-season games (27 teams x 8 games / 2), 13 conference playoff games (3 CWUAA, 5 OUA, 3 RSEQ, 2 AUS), 3 national playoff games, and 1 exhibition game (MAN vs. REG). The 2019 season was marked by a dearth of preseason games, with only one such game being played. A list of the games added to the database is available in Appendix 1.
The CFL data was scraped automatically from the league website (CFL n.d.), but instead of using the existing VBA scraper, which had become ineffective due to certain changes in the league website, a new scraper was developed in Python using Selenium (Software Freedom Conservancy 2019) and BeautifulSoup (Richardson 2007), although the basic functioning and output did not change. Cleanup was again done with simple VBA scripts, the CFL data is considerably more standardized, as it is derived from the CFL GSIS system with a standard nomenclature. A list of the games added is available in Appendix 2. 95 total games were added, 9 pre-season games, 81 regular season games, and 5 playoff games.
Existing models for P(1D) (Clement 2018b, [d] 2018) and EP (Clement 2019, 2020) were recalculated with no meaningful difference in the results, the conclusions of prior analyses have not been disturbed.
4. Conclusion
Ongoing addition of play-by-play data to the Passes & Patterns database is important in order to allow for better granularity and richer datasets to make further insights into the nature of the game. As the leading font of research into U Sports football it is hoped that this work will inspire others to make their own forays into the field. The advancement of any scientific endeavour requires a chorus of many voices and the promotion of healthy dialogue.
5. References
AUS. n.d. “AUS Home Page.” Atlantic University Sport. Accessed June 10, 2018. http://www.atlanticuniversitysport.com/landing/index.
CFL. n.d. “CFL.ca - Official Site of the Canadian Football League.” CFL.ca. Accessed August 26, 2018. https://www.cfl.ca/.
Clement, Christopher M. 2018a. “It’s the Data, Stupid: Development of a U Sports Football Database.” Passes and Patterns. June 30, 2018. http://passesandpatterns.blogspot.com/2018/06/its-data-stupid-development-of-u-sports.html.
———. 2018b. “Three Downs Away: P(1D) In U Sports Football.” Passes & Patterns. August 23, 2018. http://passesandpatterns.blogspot.com/2018/08/three-downs-away-p1d-in-u-sports.html.
———. 2018c. “Going Pro: Developing a CFL Play-by-Play Database.” Passes & Patterns. August 30, 2018. https://passesandpatterns.blogspot.com/2018/08/going-pro-developing-cfl-play-by-play.html.
———. 2018d. “The Whole Ten Yards: P(1D) in the CFL.” Passes & Patterns. September 17, 2018. https://passesandpatterns.blogspot.com/2018/09/the-whole-ten-yards-p1d-in-cfl.html.
———. 2019. “It’s Spelt ‘Rouge:’ Expected Points in U Sports.” Passes & Patterns. February 21, 2019. https://passesandpatterns.blogspot.com/2019/02/its-spelt-rouge-expected-points-in-u_21.html.
———. 2020. “Reverting to the Mean: Regression EP Models in U Sports Football.” Passes & Patterns. January 3, 2020. https://passesandpatterns.blogspot.com/2020/01/reverting-to-mean-regression-ep-models.html.
CWUAA. n.d. “CWUAA Home Page.” Canada West Universities Athletic Association. Accessed June 10, 2018. https://www.canadawest.org/landing/index.
OUA. n.d. “OUA Home Page.” Ontario University Athletics. Accessed June 10, 2018. http://www.oua.ca/landing/index.
PrestoSports. n.d. “PrestoSports Home Page.” PrestoSports. Accessed June 10, 2018. https://www.prestosports.com/landing/index.
Richardson, Leonard. 2007. “Beautiful Soup 4.4.0 Documentation.” Beautiful Soup. April 2007. https://www.crummy.com/software/BeautifulSoup/bs4/doc/.
RSEQ. n.d. “RSEQ Home Page.” Réseau Du Sport étudiant Du Québec. Accessed June 10, 2018. http://rseq.ca/.
Software Freedom Conservancy. 2019. “SeleniumHQ Browser Automation.” Selenium. December 10, 2019. https://selenium.dev/.
The Automated Scorebook. n.d. “The Automated Scorebook.” The Automated Scorebook. Accessed June 18, 2018. http://www.automatedscorebook.com/.
6. Appendix 1: List of U Sports games
BIS vs. ACA 2019-08-24 1505 Raymond Field
TOR vs. WAT 2019-08-25 1904 Warrior Field
CAR vs. QUE 2019-08-25 1800 New Richardson Memorial Stadium
MAC vs. GUE 2019-08-25 1300 Guelph Alumni Stadium
WES vs. WLU 2019-08-25 1304 University Stadium
YRK vs. WIN 2019-08-25 1900 University of Windsor Stadium
MAN vs. REG 2019-08-21 1800 Mosaic Stadium
SKH vs. MAN 2019-08-30 1900 Investors Group Field
CGY vs. ALB 2019-08-30 1900 Foote Field
REG vs. UBC 2019-08-31 1905 Thunderbird Stadium
OTT vs. MAC 2019-09-01 1604 Ron Joyce Stadium
WIN vs. TOR 2019-09-01 1902 Varsity Stadium
GUE vs. CAR 2019-09-01 1930 MNP Park
WLU vs. YRK 2019-09-02 1307 York Lions Stadium
QUE vs. WES 2019-09-02 1930 TD Waterhouse Stadium
SMU vs. MTA 2019-08-31 1405 MacAulay Field
SFX vs. BIS 2019-08-31 1905 Coulter Field
MAN vs. CGY 2019-09-09 1737 McMahon Stadium
UBC vs. SKH 2019-09-09 1905 Griffiths Stadium
ALB vs. REG 2019-09-09 1905 Mosaic Stadium
YRK vs. GUE 2019-09-07 1300 Guelph Alumni Stadium
WAT vs. WIN 2019-09-07 1302 University of Windsor Stadium
QUE vs. OTT 2019-09-07 1304 Gee-Gees Field
WES vs. MAC 2019-09-07 1304 Ron Joyce Stadium
TOR vs. WLU 2019-09-07 1904 University Stadium
BIS vs. SMU 2019-09-06 1905 Huskies Stadium
ACA vs. SFX 2019-09-06 1900 Oland Stadium
MAN vs. ALB 2019-09-13 1900 Foote Field
SKH vs. REG 2019-09-13 1900 Mosaic Stadium
CGY vs. UBC 2019-09-14 1500 Thunderbird Stadium
REG vs. CGY 2019-09-20 1900 McMahon Stadium
SKH vs. ALB 2019-09-21 1300 Foote Field
UBC vs. MAN 2019-09-21 1400 Investors Group Field
WLU vs. OTT 2019-09-14 1304 Gee-Gees Field
GUE vs. WAT 2019-09-14 1304 Warrior Field
MAC vs. YRK 2019-09-14 1304 York Lions Stadium
WIN vs. QUE 2019-09-14 1300 New Richardson Memorial Stadium
CAR vs. WES 2019-09-14 1900 TD Waterhouse Stadium
WES vs. GUE 2019-09-21 1300 Guelph Alumni Stadium
WIN vs. MAC 2019-09-21 1304 Ron Joyce Stadium
QUE vs. TOR 2019-09-21 1306 Varsity Stadium
WAT vs. CAR 2019-09-21 1300 MNP Park
OTT vs. YRK 2019-09-21 1704 York Lions Stadium
MTA vs. ACA 2019-09-21 1405 Raymond Field
SMU vs. SFX 2019-09-21 1405 Oland Stadium
MTA vs. BIS 2019-09-14 1405 Coulter Field
ACA vs. SMU 2019-09-13 1905 Huskies Stadium
CGY vs. SKH 2019-09-27 1900 Griffiths Stadium
REG vs. MAN 2019-09-28 1400 Investors Group Field
ALB vs. UBC 2019-09-28 1507 Thunderbird Stadium
YRK vs. WAT 2019-09-28 1304 Warrior Field
WES vs. WIN 2019-09-28 1302 University of Windsor Stadium
OTT vs. GUE 2019-09-28 1300 Guelph Alumni Stadium
TOR vs. CAR 2019-09-28 1300 MNP Park
QUE vs. WLU 2019-09-28 1304 University Stadium
ACA vs. BIS 2019-09-28 1405 Coulter Field
SFX vs. MTA 2019-09-28 1405 MacAulay Field
MAN vs. SKH 2019-10-04 1900 Griffiths Stadium
UBC vs. REG 2019-10-04 1905 Mosaic Stadium
ALB vs. CGY 2019-10-05 1307 McMahon Stadium
CAR vs. OTT 2019-10-05 1204 TD Place
WLU vs. WIN 2019-10-05 1302 University of Windsor Stadium
WAT vs. WES 2019-10-05 1300 TD Waterhouse Stadium
MAC vs. TOR 2019-10-05 1304 Varsity Stadium
BIS vs. SFX 2019-10-04 1905 Oland Stadium
MTA vs. SMU 2019-10-04 1901 Huskies Stadium
GUE vs. QUE 2019-10-10 1800 New Richardson Memorial Stadium
WLU vs. WAT 2019-10-10 1904 Warrior Field
MAC vs. CAR 2019-10-11 1300 MNP Park
TOR vs. YRK 2019-10-11 1914 York Lions Stadium
WIN vs. OTT 2019-10-11 1904 Gee-Gees Field
SMU vs. BIS 2019-10-11 1905 Coulter Field
SFX vs. ACA 2019-10-11 1900 Raymond Field
MTA vs. SMU 2019-10-04 1901 Huskies Stadium
BIS vs. SFX 2019-10-04 1905 Oland Stadium
SFX vs. ACA 2019-10-11 1900 Raymond Field
SMU vs. BIS 2019-10-11 1905 Coulter Field
SMU vs. ACA 2019-10-19 1400 Raymond Field
BIS vs. MTA 2019-10-19 1505 MacAulay Field
SFX vs. SMU 2019-10-26 1405 Huskies Stadium
ACA vs. MTA 2019-10-26 1400 MacAulay Field
MON vs. ACA 2019-11-16 1407 Raymond Field
CAR vs. WLU 2019-10-18 1104 University Stadium
GUE vs. TOR 2019-10-19 1306 Varsity Stadium
YRK vs. QUE 2019-10-19 1300 New Richardson Memorial Stadium
OTT vs. WES 2019-10-19 1300 TD Waterhouse Stadium
WAT vs. MAC 2019-10-19 1304 Ron Joyce Stadium
MAC vs. WES 2019-11-09 1315 TD Waterhouse Stadium
MAC vs. CGY 2019-11-16 1307 McMahon Stadium
REG vs. ALB 2019-10-19 1300 Foote Field
SKH vs. UBC 2019-10-19 1307 Thunderbird Stadium
CGY vs. MAN 2019-10-19 1400 Investors Group Field
MAN vs. REG 2019-10-25 1900 Mosaic Stadium
UBC vs. CGY 2019-10-26 1507 McMahon Stadium
ALB vs. SKH 2019-10-26 1400 Griffiths Stadium
MAN vs. CGY 2019-11-02 1206 McMahon Stadium
ALB vs. SKH 2019-11-02 1500 Griffiths Stadium
SKH vs. CGY 2019-11-09 1307 McMahon Stadium
CGY vs. MON 2019-11-23 1316 CEPSUM
MON vs. ACA 2019-11-16 1407 Raymond Field
MTA vs. SFX 2019-08-24 1405 Oland Stadium
MTA vs. BIS 2019-11-02 1808 Coulter Field
BIS vs. ACA 2019-11-09 1405 Raymond Field
WAT vs. OTT 2019-10-26 1304 Gee-Gees Field
CAR vs. GUE 2019-10-26 1300 Guelph Alumni Stadium
WAT vs. WES 2019-11-02 1300 TD Waterhouse Stadium
GUE vs. MAC 2019-11-02 1304 Ron Joyce Stadium
SHE vs. LAV 2019-08-25 1300 Stade TELUS
MON vs. CON 2019-08-23 1900 Concordia Stadium
SHE vs. MON 2019-08-30 1900 CEPSUM
CON vs. MCG 2019-08-30 1600 Percival Molson Stadium
MON vs. MCG 2019-09-07 1310 Percival Molson Stadium
LAV vs. CON 2019-09-07 1407 Concordia Stadium
MCG vs. LAV 2019-09-22 1300 Stade TELUS
SHE vs. CON 2019-09-21 1300 Concordia Stadium
LAV vs. MON 2019-09-14 1400 CEPSUM
MCG vs. SHE 2019-09-13 1900 Stade de l'Universite
SHE vs. MCG 2019-09-28 1307 Percival Molson Stadium
CON vs. MON 2019-09-27 1911 CEPSUM
MCG vs. MON 2019-10-04 1900 CEPSUM
CON vs. LAV 2019-10-06 1309 Stade TELUS
LAV vs. MCG 2019-10-12 1310 Percival Molson Stadium
MON vs. SHE 2019-10-12 1400 Stade de l'Universite
CON vs. SHE 2019-10-19 1600 Stade de l'Universite
MON vs. LAV 2019-10-20 1300 Stade TELUS
MCG vs. CON 2019-10-26 1400 Concordia Stadium
LAV vs. SHE 2019-10-26 1400 Stade de l'Universite
CON vs. LAV 2019-11-02 1300 Stade TELUS
MCG vs. MON 2019-11-02 1400 CEPSUM
MON vs. LAV 2019-11-09 1400 Stade TELUS
7. Appendix 2: List of CFL games
LNS vs. ESK 2019-05-26 1400 Commonwealth Stadium
ALS vs. ARG 2019-05-30 Varsity Stadium
ESK vs. BBO 2019-05-31 1930 Investors Group Field
RRI vs. STA 2019-05-31 1900 McMahon Stadium
TIC vs. RED 2019-06-01 1900 TD Place
ARG vs. TIC 2019-06-06 1930 Tim Hortons Field
RED vs. ALS 2019-06-06 1930 Percival Molson Stadium
BBO vs. RRI 2019-06-06 1900 Taylor Field at Mosaic Stadium
STA vs. LNS 2019-06-07 1900 BC Place
RRI vs. TIC 2019-06-13 1930 Tim Hortons Field
ALS vs. ESK 2019-06-14 1900 Commonwealth Stadium
RED vs. STA 2019-06-15 1700 McMahon Stadium
BBO vs. LNS 2019-06-15 1900 BC Place
RRI vs. RED 2019-06-20 1930 TD Place
LNS vs. ESK 2019-06-21 1900 Commonwealth Stadium
TIC vs. ARG 2019-06-22 1900 BMO Field
ESK vs. BBO 2019-06-27 1930 Investors Group Field
ALS vs. TIC 2019-06-28 1930 Tim Hortons Field
LNS vs. STA 2019-06-29 1700 McMahon Stadium
ARG vs. RRI 2019-07-01 1700 Taylor Field at Mosaic Stadium
TIC vs. ALS 2019-07-04 1930 Percival Molson Stadium
BBO vs. RED 2019-07-05 1930 TD Place
LNS vs. ARG 2019-07-06 1900 BMO Field
STA vs. RRI 2019-07-06 2000 Taylor Field at Mosaic Stadium
ESK vs. LNS 2019-07-11 1900 BC Place
ARG vs. BBO 2019-07-12 1930 Investors Group Field
ALS vs. RED 2019-07-13 1600 TD Place
STA vs. TIC 2019-07-13 1900 Tim Hortons Field
ARG vs. STA 2019-07-18 1900 McMahon Stadium
RED vs. BBO 2019-07-19 1930 Investors Group Field
ESK vs. ALS 2019-07-20 1600 Percival Molson Stadium
LNS vs. RRI 2019-07-20 1700 Taylor Field at Mosaic Stadium
STA vs. RED 2019-07-25 1900 TD Place
ARG vs. ESK 2019-07-25 1930 Commonwealth Stadium
BBO vs. TIC 2019-07-26 1900 Tim Hortons Field
RRI vs. LNS 2019-07-27 1600 BC Place
BBO vs. ARG 2019-08-01 1900 BMO Field
TIC vs. RRI 2019-08-01 1930 Taylor Field at Mosaic Stadium
RED vs. ALS 2019-08-02 1900 Percival Molson Stadium
ESK vs. STA 2019-08-03 1700 McMahon Stadium
STA vs. BBO 2019-08-08 1930 Investors Group Field
RRI vs. ALS 2019-08-09 1900 Percival Molson Stadium
RED vs. ESK 2019-08-09 2000 Commonwealth Stadium
LNS vs. TIC 2019-08-10 1900 Tim Hortons Field
LNS vs. BBO 2019-08-15 1930 Investors Group Field
ESK vs. ARG 2019-08-16 1930 BMO Field
TIC vs. RED 2019-08-17 1600 TD Place
ALS vs. STA 2019-08-17 1700 McMahon Stadium
BBO vs. ESK 2019-08-23 1900 Commonwealth Stadium
RED vs. RRI 2019-08-24 1700 Taylor Field at Mosaic Stadium
TIC vs. LNS 2019-08-24 1900 BC Place
ALS vs. ARG 2019-08-25 1830 BMO Field
BBO vs. RRI 2019-09-01 1300 Taylor Field at Mosaic Stadium
ARG vs. TIC 2019-09-02 1300 Tim Hortons Field
ESK vs. STA 2019-09-02 1430 McMahon Stadium
LNS vs. ALS 2019-09-06 1930 Percival Molson Stadium
ARG vs. RED 2019-09-07 1300 TD Place
RRI vs. BBO 2019-09-07 1500 Investors Group Field
STA vs. ESK 2019-09-07 1700 Commonwealth Stadium
RED vs. LNS 2019-09-13 1900 BC Place
TIC vs. STA 2019-09-14 1400 McMahon Stadium
ALS vs. RRI 2019-09-14 1700 Taylor Field at Mosaic Stadium
STA vs. ARG 2019-09-20 1900 BMO Field
TIC vs. ESK 2019-09-20 1930 Commonwealth Stadium
BBO vs. ALS 2019-09-21 1600 Percival Molson Stadium
LNS vs. RED 2019-09-21 1900 TD Place
TIC vs. BBO 2019-09-27 1930 Investors Group Field
ESK vs. RED 2019-09-28 1600 TD Place
RRI vs. ARG 2019-09-28 1900 BMO Field
ALS vs. LNS 2019-09-28 1900 BC Place
ESK vs. TIC 2019-10-04 1900 Tim Hortons Field
STA vs. ALS 2019-10-05 1600 Percival Molson Stadium
BBO vs. RRI 2019-10-05 1700 Taylor Field at Mosaic Stadium
ARG vs. LNS 2019-10-05 1900 BC Place
RED vs. ARG 2019-10-11 1900 BMO Field
RRI vs. STA 2019-10-11 1930 McMahon Stadium
ALS vs. BBO 2019-10-12 1500 Investors Group Field
LNS vs. ESK 2019-10-12 1700 Commonwealth Stadium
ARG vs. ALS 2019-10-18 1900 Moncton Stadium
RRI vs. LNS 2019-10-18 1900 BC Place
RED vs. TIC 2019-10-19 1600 Tim Hortons Field
BBO vs. STA 2019-10-19 1700 McMahon Stadium
STA vs. BBO 2019-10-25 1930 Investors Group Field
TIC vs. ALS 2019-10-26 1300 Percival Molson Stadium
RED vs. ARG 2019-10-26 1600 BMO Field
RRI vs. ESK 2019-10-26 1700 Commonwealth Stadium
ALS vs. RED 2019-11-01 1900 TD Place
ESK vs. RRI 2019-11-02 1500 Taylor Field at Mosaic Stadium
ARG vs. TIC 2019-11-02 1900 Tim Hortons Field
STA vs. LNS 2019-11-02 1900 BC Place
ESK vs. ALS 2019-11-10 1300 Percival Molson Stadium
BBO vs. STA 2019-11-10 1330 McMahon Stadium
ESK vs. TIC 2019-11-17 1300 Tim Hortons Field
BBO vs. RRI 2019-11-17 1330 Taylor Field at Mosaic Stadium
TIC vs. BBO 2019-11-24 1800 McMahon Stadium
No comments:
Post a Comment