41

It feels like there is a plague of people posting screenshots of data in SO questions. Perhaps it would be possible to add a basic classification model to the image upload functionality that alerts users who appear to be uploading pictures of data to stop and enter the data directly into the question instead.

Edit: I have done some work to explore whether this would be possible, and it appears it might be.

TLDR: It is possible to detect when tabular data is posted as an image with a fairly high degree of accuracy.

Example of the Problem

A user wants help transforming data, but pastes a screenshot instead of the actual data. First comment is asking the user to please post the data, not a photo. This happens with code, tracebacks, queries, etc.

Scope

The exploratory work is limited to SO posts tagged 'python' in an effort to detect users who are posting images of tabular data instead of pasting it as text. Other tags would have similar problems, like SQL queries and so forth, which as such are excluded for this attempt.

Gather Data

I used the Data Explorer to pull down a sample of posts with the python tag that were at least 90+ days old. Using regex to extract image links from the post body, I ended up with about 2500 sample images.

The images were sorted by hand into a data/not_data classes. After initial sorting about 17% of the images had some sort of tabular data in them. Since the data images were a significant minority they were oversampled during training to even out the class balance.

Here is a sample of the images as they were fed into the model:

Sample of sorted images

Modeling

The model was trained using resnet101 and was able to achieve a validation error of .071 with minimal image augmentation (hflip, scaling)

Results

I pulled additional python tagged posts within the last 90 days and extracted images again. Each image was ran through the classifier and the prediction was captured.

With the fresh data, we see 12% of posts having images, with 20% of those images being classified as having tabular data.

Here are some samples of the images that were classified:

Classified as Having Tabular Data Tabular Data Predictions

Classified as NOT Having Tabular Data Other Data Predictions

Other Ideas A manual review of the predictions results is very promising. Of course additional classes or other binary class models would need to be built to be practical for other uses.

I had also considered where a reputation cutoff could be useful, such that only users below a certain rep would need to have his/her images classified. It could be useful to not bug more experienced users, however there are examples of high-reputation users exhibiting this behavior:

High-Rep User Posting Images of Data

In any case the rep of users posting data images is generally lower than others.

Rep Stats of Users who posted images of tabular data (predicted)

Statistic Value
count 1204.000000
mean 651.740864
std 4262.234174
min 1.000000
25% 19.000000
50% 57.000000
75% 217.000000
max 96408.000000

Rep Stats of Users who posted images of non-tabular data (predicted)

Statistic Value
count 4761.000000
mean 943.694182
std 6570.978536
min 1.000000
25% 17.000000
50% 65.000000
75% 352.000000
max 257582.000000
5
  • 4
    Even better (on some sites, at least) would be detecting that a screen shot as a lot of text in it and encourage the poster to include code, input, and/or output as text, and not as an image. Dec 20 '20 at 19:05
  • 1
    Chris, it's always good to include a few examples (link to post); this allows one to judge the quality of the images (machine or only human readable) and assess the complexity of the request.
    – Rob
    Dec 20 '20 at 21:19
  • 1
    I'll be happy to train a model and see what feasibility is. Good to know if found useful it would have a chance to be implemented.
    – Chris
    Dec 20 '20 at 21:43
  • 1
    same goes for screen shots of logs/error messages/code
    – Shai
    Dec 22 '20 at 6:25
  • Screenshots of code are popular as well.
    – Mast
    Aug 7 at 20:21

You must log in to answer this question.

Browse other questions tagged .