GSoC 2017: Work Summary

So finally Google Summer of Code 2017 is coming to an end. It was an amazing experience to be a part of this prestigious program and I learned a lot in these three months about open source development.

Work Summary

I worked on dateparser, a project under sub-org Scrapinghub of Python Software Foundation (PSF), that deals with parsing dates in various languages and formats. The objective of the project was to integrate translation data of all locales in Unicode Common Locale Data Repository(CLDR) which is a standard repository of locale specific data with the existing translation data in dateparser. Here is a brief outline of the work done on my project on dateparser during GSoC 2017:

Work Completed

1. Retrieved Translation data from Unicode CLDR
Scripts were written to retrieve translation data from unicode CLDR github repository. The translation data for dates and numerals were separately stored.

2. Ordered languages by population
The languages were ordered on the basis of the population that use the languages using data contained in cldr-core/supplemental/territoryInfo.json.

 3. Integration with Existing Translation data
The data retrieved from unicode CLDR was integrated with existing translation data that has been contributed by the community.

4. Modifying Codebase to work with Integrated Translation data
The data retrieved from unicode CLDR was stored in a different format than previous data, and thus it was required to make changes in the codebase to work with the new integrated database.

5. Adding Support for Locales
Changes were made in the codebase to support parsing with locales.

The entire work done during GSoC 2017 is contained in PR #321 (last commit).

Work to be done

1. Adding Support for Numeral Words 
Initially I also planned to add support for dates with numeral words during the GSoC period itself but could not implement it. I added numeral translation data for translating numerals and I planned on implementing a numeral parser to parse numeral strings, and further using it to parse dates with numeral words. I will start working on numeral parser soon after the GSoC period ends.

2. Adding more tests
With addition of translation data of more than 200 languages and over 500 locales, a lot of tests need to be added. Though I have already added many tests, I will add more in future. 

Popular posts from this blog

Integrate Unicode CLDR: Working with Retrieved Data

I have been working on scripts and modifying data for quite some time and it seems the data is ready to be used for parsing dates. The data retrieved from Unicode CLDR  has been divided into two parts: numeral_translation_data , that will be specifically used to parse numerals, and date_translation_data ,   that will be used to parse date strings after modifying it by parsing numerals if included in the date string. The existing data that was contributed by various individuals to dateparser has been modified to supplement the data retrieved from CLDR, i.e., only the portion of data that is not included in data retrieved from CLDR remains as supplementary data , which will continue to be modified by contributors in future. The data for date translation consists of translations for months, weekdays and periods, date order, and translations for relative-type dates. This data is stored for different languages and for each language, locale-specific data is stored for all locales with th
Read more

GSoC 2017 : My Project

So I have been finally selected for Google Summer of Code 2017  at Python Software Foundation . Python Software Foundation(PSF) serves as an umbrella organisation comprising various sub-orgs that have projects that contribute to the development of the Python language. The sub-org I am going to work with is Scrapinghub . About Scrapinghub Scrapinghub, as its name clearly suggests, deals primarily with scraping the web. Scrapinghub is a company concerned with Information Retrieval and its later manipulation, i.e., it deals with both data extraction and data processing after retrieval. It has various projects that deal with these topics and the one I am going to work on deals with data processing after retrieval. I am going to work with dateparser , which is a Python library that primarily deals with parsing dates in various languages and formats. About dateparser dateparser is a Python library that is used to parse various forms of dates in different languages to a common format.
Read more