OONI-dev meeting and hackathon 2016
Oonitarians are spread out across the globe, and OONI’s core team is no exception. We recently met in Berlin to hack on OONI’s software, work on our roadmap, and to have all those discussions that are more fruitful to have in person than on IRC. We also held a hackathon to engage new individuals with our project!
Below we document some of the things that we worked on as part of our OONI-dev meeting and hackathon.
OONI’s internal meeting was an opportunity for the project’s core team members to meet in person, discuss priorities over the next months, and to adjust the project’s roadmap accordingly. This meeting took place between Monday, 7th November and Thursday, 10th November.
We started off by mapping all of the ideas, questions, and problems that we wanted to collaboratively discuss and work on as part of the meeting. We then proceeded with voting amongst us on the ones that we believed were of highest priority, and subsequently creating sessions out of them.
As a result, we spent the first two days of the OONI-dev meeting having discussions as part of the following six sessions.
Session 1: OONI Explorer user stories.
Session 2: Probe orchestration.
Session 4: UX design.
Session 6: OONI mobile client.
We then spent the third day working on our roadmap for the next six months, adjusting it to the project’s needs and priorities based on the discussions from the sessions of the previous days. During the fourth (and final) day of our internal meeting we held smaller, ad hoc discussions based on the rest of the ideas/questions/problems that were mapped out in the first day, and which didn’t fit into any of the six core sessions of the first two days. We also spent the last day of the internal meeting hacking on ooniprobe and measurement-kit (a network measurement library that ooniprobe is being integrated in).
The OONI hackathon, held at Berlin’s OnionSpace on 11th and 12th November, was an opportunity for us to meet new people interested in engaging with our project. We were excited to see that individuals with diverse skills and backgrounds showed up at the hackathon, including data scientists, programmers, researchers, and activists from various countries around the world.
Below is a summary of the things that we worked on as part of the two-day hackathon.
OONI Explorer provides a web interface to explore all of OONI’s network measurements. However, OONI’s data processing pipeline still needs some improvement to make OONI’s data more accessible and easier to navigate. As such, we were thrilled to have data scientists join the hackathon and offer suggestions on how to improve our data analysis techniques.
We discussed and defined queries for data analysis that can help improve the next iteration of OONI’s pipeline. This included:
Defining heuristics for separating “normal” from “anomalous” measurements as part of OONI’s data processing;
Filtering the web connectivity “anomalous” measurements based on the types of anomalies (DNS, HTTP-diff, HTTP-failure, TCP/IP);
Aggregating the sum of anomalies per measurement;
Filtering measurements based on their testing frequency.
Furthermore, we discussed and implemented techniques for identifying and filtering false positives that emerge in the measurements. As part of this, we discussed ways to filter Cloudflare false positives, as well as techniques for better identifying block pages within measurements. We also discussed a methodology for detecting URLs that fail in the same way across measurements, across time, and across countries, and filtering them as false positives.
OONI’s measurement API
In addition to OONI Explorer, OONI’s network measurements are published in JSON format. One of our priorities though is to also build an API to the measurements that enables data analysis, allowing users to query the API to retrieve measurements based on certain characteristics (to query, for example, all anomalous measurements).
As part of the hackathon, we continued our work on OONI’s new measurement API and created a staging version of it.
Several web developers joined the hackathon and we worked with them on the following:
Restructuring the “Tests” section of the website. OONI has developed multiple free software tests and this section of the website was created to not only list them, but to also provide descriptions explaining what each test is designed to do. As part of the hackathon, one of the participants worked on restructuring this section of the website to improve the accessibility of the test descriptions.
Improving upon the presentation of information on OONI’s home page. Another participant worked on fixing some bugs and restructuring the home page of OONI’s website to make the flow of information more accessible.
Integrating a Frequently Asked Questions (FAQ) section on the website. Having a page on OONI’s website which addresses some of the questions that we frequently get asked by community members is important. As part of the hackathon, we worked on creating a FAQ section for OONI’s website. If any important questions are missing in there, please don’t hesitate to contact us.
Measurement-kit is a network measurement library that ooniprobe is gradually being integrated in. As part of the hackathon, we worked on improving the DNS capabilities of measurement-kit along the following lines:
We developed fixes for several issues that prevent the default, libevent-based DNS engine from working correctly with IPv6 in certain cases;
We started integrating a new DNS engine that uses the system’s DNS functionality;
We started writing a new DNS engine based on the c-ares DNS library.
In addition, we also started implementing support for cloud-fronted collectors in measurement-kit to allow users to submit the result of OONI measurements to our servers through a cloud-based service (e.g. Amazon), thus masking the fact that they are communicating with OONI servers. We also committed enhancements and fixes to support the development of mobile applications based on measurement-kit.
Spoiler: Over the next few months we will be releasing a mobile app (for Android and iOS) for running ooniprobe! During the hackathon, we worked on improving the design of OONI’s mobile app.
Here’s a sneak peak of OONI’s upcoming mobile app:
Screenshot: OONI mobile
Screenshot: OONI mobile testing http://neubot.org
An important part of identifying censorship is determining which websites to examine for blocking.
OONI bases its testing on the Citizen Lab’s test list Github repository which includes a wide range of country-specific lists of URLs. These lists constantly need to be updated with new URLs and, in many cases, new lists need to be created for countries that don’t already have their own test lists.
We were happy to see that various participants joined the hackathon to help contribute to existing test lists and to learn how to create new ones. As part of the hackathon, participants worked on updating the Palestinian test list to include many new URLs. We also prioritized on updating test lists for countries with upcoming elections, such as the test list for Gambia.
Amongst the new faces that joined us at the hackathon, we were also happy to meet people curious to learn about OONI and how to run ooniprobe. We discussed OONI’s approaches and methodologies, and taught participants how to run various ooniprobe software tests. As an outcome, we hope that participants will share the skills and knowledge they acquired through the hackathon with others too, and help increase transparency on internet censorship around the world.
The OONI team would like to send a warm thank you to all the participants who joined the hackathon, to the OnionSpace for sharing its space with us for a week, and to the Ford Foundation for covering some of the OONI team’s travel costs, thus making this meeting and hackathon possible.