It feels like there is a plague of people posting screenshots of data in SO questions. Perhaps it would be possible to add a basic classification model to the image upload functionality that alerts users who appear to be uploading pictures of data to stop and enter the data directly into the question instead.
Edit: I have done some work to explore whether this would be possible, and it appears it might be.
TLDR: It is possible to detect when tabular data is posted as an image with a fairly high degree of accuracy.
A user wants help transforming data, but pastes a screenshot instead of the actual data. First comment is asking the user to please post the data, not a photo. This happens with code, tracebacks, queries, etc.
Scope
The exploratory work is limited to SO posts tagged 'python' in an effort to detect users who are posting images of tabular data instead of pasting it as text. Other tags would have similar problems, like SQL queries and so forth, which as such are excluded for this attempt.
Gather Data
I used the Data Explorer to pull down a sample of posts with the python tag that were at least 90+ days old. Using regex to extract image links from the post body, I ended up with about 2500 sample images.
The images were sorted by hand into a data / not_data classes. After initial sorting about 17% of the images had some sort of tabular data in them. Since the data images were a significant minority they were oversampled during training to even out the class balance.
Here is a sample of the images as they were fed into the model:
Modeling
The model was trained using resnet101 and was able to achieve a validation error of .071 with minimal image augmentation (hflip, scaling)
Results
I pulled additional python tagged posts within the last 90 days and extracted images again. Each image was ran through the classifier and the prediction was captured.
With the fresh data, we see 12% of posts having images, with 20% of those images being classified as having tabular data.
Here are some samples of the images that were classified:
Classified as Having Tabular Data
Classified as NOT Having Tabular Data
Other Ideas A manual review of the predictions results is very promising. Of course additional classes or other binary class models would need to be built to be practical for other uses.
I had also considered where a reputation cutoff could be useful, such that only users below a certain rep would need to have his/her images classified. It could be useful to not bug more experienced users, however there are examples of high-reputation users exhibiting this behavior:
High-Rep User Posting Images of Data
In any case the rep of users posting data images is generally lower than others.
Rep Stats of Users who posted images of tabular data (predicted)
count 1204.000000
mean 651.740864
std 4262.234174
min 1.000000
25% 19.000000
50% 57.000000
75% 217.000000
max 96408.000000
Rep Stats of Users who posted images of non-tabular data (predicted)
count 4761.000000
mean 943.694182
std 6570.978536
min 1.000000
25% 17.000000
50% 65.000000
75% 352.000000
max 257582.000000
