It feels like there is a plague of people posting screenshots of data in SO questions. Perhaps it would be possible to add a basic classification model to the image upload functionality that alerts users who appear to be uploading pictures of data to stop and enter the data directly into the question instead.
Edit: I have done some work to explore whether this would be possible, and it appears it might be.
TLDR: It is possible to detect when tabular data is posted as an image with a fairly high degree of accuracy.
A user wants help transforming data, but pastes a screenshot instead of the actual data. First comment is asking the user to please post the data, not a photo. This happens with code, tracebacks, queries, etc.
The exploratory work is limited to SO posts tagged 'python' in an effort to detect users who are posting images of
tabular data instead of pasting it as text. Other tags would have similar problems, like
SQL queries and so forth, which as such are excluded for this attempt.
I used the
Data Explorer to pull down a sample of posts with the
python tag that were at least 90+ days old. Using
regex to extract image links from the post body, I ended up with about
2500 sample images.
The images were sorted by hand into a
not_data classes. After initial sorting about
17% of the images had some sort of tabular data in them. Since the data images were a significant minority they were oversampled during training to even out the class balance.
Here is a sample of the images as they were fed into the model:
The model was trained using
resnet101 and was able to achieve a validation error of
.071 with minimal image augmentation (
I pulled additional
python tagged posts within the last 90 days and extracted images again. Each image was ran through the classifier and the prediction was captured.
With the fresh data, we see
12% of posts having images, with
20% of those images being classified as having tabular data.
Here are some samples of the images that were classified:
Other Ideas A manual review of the predictions results is very promising. Of course additional classes or other binary class models would need to be built to be practical for other uses.
I had also considered where a reputation cutoff could be useful, such that only users below a certain rep would need to have his/her images classified. It could be useful to not bug more experienced users, however there are examples of high-reputation users exhibiting this behavior:
In any case the rep of users posting data images is generally lower than others.
Rep Stats of Users who posted images of tabular data (predicted)
count 1204.000000 mean 651.740864 std 4262.234174 min 1.000000 25% 19.000000 50% 57.000000 75% 217.000000 max 96408.000000
Rep Stats of Users who posted images of non-tabular data (predicted)
count 4761.000000 mean 943.694182 std 6570.978536 min 1.000000 25% 17.000000 50% 65.000000 75% 352.000000 max 257582.000000