How to process photos and videos using a neural network | Articles | Photo, video, optics

Machine learn­ing and neur­al net­works can be cool tools for pho­tog­ra­phers and artists, but inte­grat­ing them into your work­flow is a chal­lenge for those who can’t code. We have put togeth­er a selec­tion of appli­ca­tions based on neur­al net­works that are easy to under­stand and start work­ing in just a cou­ple of clicks.

Waifu2x

A free online ser­vice in which a neur­al net­work improves the qual­i­ty of a pho­to. It removes noise and increas­es the res­o­lu­tion of images by the desired num­ber of times, and also sharp­ens. Extreme­ly easy to use inter­face.

Noise sup­pres­sion exam­ple with Waifu2x. Source: overclockers.ru

Let’s Enhance

Let’s Enhance is a com­mer­cial ana­logue of Waifu2x. The pur­pose of the ser­vice is the same — it is a neur­al net­work to improve pho­tos online. You can increase the res­o­lu­tion of the pic­ture by 16 times, as well as adjust its col­or and tone and remove noise. You can upload sev­er­al images to the site at once and start pro­cess­ing them simul­ta­ne­ous­ly.

The free ver­sion allows you to increase the res­o­lu­tion by 4 times for only five pho­tos. The sub­scrip­tion allows you to increase the res­o­lu­tion by 16 times for 100 images and costs $9 per month. For pro­cess­ing 300 pho­tos, you will have to pay $24, and for 500 — $34.

An exam­ple of how Let’s Enhance algo­rithms work. Source: letsenhance.io

Artbreeder

An image gen­er­a­tion ser­vice with the fol­low­ing func­tions:

1. “Gen­er­al” (Gen­er­al) — allows you to cross any image with each oth­er. Noth­ing pre­vents you from cross­ing a piz­za and a par­rot, but the result will be abstract and far from pho­to­re­al­ism.

2. “Por­traits” (Por­traits) — allows you to col­lect the por­trait you need from sev­er­al images of faces. Faces have cus­tomiz­able options. You can set the physique, age, gen­der, mood, realistic/drawn facial out­lines, and so on.

3. “Album Cov­ers” (Albums) is the gen­er­a­tion of abstract images in the man­ner of music album cov­ers.

There are also func­tions for gen­er­at­ing land­scapes, paint­ings, build­ings and ani­mat­ed char­ac­ters.

In order to use Art­breed­er, you will have to stand in line or pay a tar­iff, the cheap­est of which costs about 700 rubles. The paid plan will allow you to save the gen­er­a­tion results to your com­put­er at a res­o­lu­tion of 2048x2048 (in the free plan, the res­o­lu­tion is 4 times low­er, and you can only save the first 4 gen­er­at­ed images), as well as gen­er­ate images based on those that you upload to the plat­form.

These land­scapes can be gen­er­at­ed using Art­breed­er. Source: deity­rox

Nvidia InPainting

Helps you retouch your pho­to with the smart brush. With the help of Nvidia InPaint­ing, you can auto­mate the design rou­tine of remov­ing unwant­ed objects from a pho­to. So, if visu­al garbage got into the frame or you need to remove an object stick­ing out of your pock­et or cor­rect skin defects, the appli­ca­tion will cope with every­thing. The algo­rithm ana­lyzes the pix­els locat­ed around the select­ed object and tries to replace them in the most appro­pri­ate way.

On the left is the orig­i­nal pho­to, on the right is the result of the work of Nvidia algo­rithms, with the help of which one of the char­ac­ters in the pho­to was cut out. Source: www.esearchgate.net

Remove.bg

If in the pre­vi­ous ser­vice we removed objects from a pho­to, now we remove the back­ground around a cer­tain object. There is a sim­i­lar func­tion in Adobe Pho­to­shop, the Adobe Sen­sei neur­al net­work is respon­si­ble for it — we wrote more about this in a sep­a­rate arti­cle.

There are ver­sions for Win­dows / MacOS / Lin­ux, plu­g­ins for Adobe Pho­to­shop, Fig­ma and oth­er pro­grams. The free ver­sion of the appli­ca­tion great­ly lim­its the res­o­lu­tion of the down­loaded result, mak­ing it unsuit­able for pro­fes­sion­al use (unless, of course, you are going to increase the res­o­lu­tion of one of the free neur­al net­works men­tioned above in the next step). The cost of the cheap­est sub­scrip­tion is 579 rubles. For this price, the ser­vice allows you to edit and down­load high-res­o­lu­tion results for 40 pho­tos.

GFP-GAN

An algo­rithm that details blur­ry faces in a pho­to. Rel­e­vant for cas­es where the shoot­ing took place in poor light­ing or “on the go”. The algo­rithm recov­ers well even very com­pressed images. Although it hap­pens that it draws unnec­es­sary details or makes the image too car­toony. You can get acquaint­ed with the work of the algo­rithm by the link.

Once you have fol­lowed the link, place the image you want into the GFP-GAN work­space. Then click Sub­mit and wait about 5 sec­onds for the algo­rithm to com­plete its work. The result and the orig­i­nal will appear in one win­dow.

On the left is the orig­i­nal pho­to, on the right is the result of GFP-GAN.

Prisma

Appli­ca­tion for Android and iOS (there is also a web ver­sion) that allows you to trans­fer the style of famous artists to pho­tos. And this does not hap­pen by apply­ing fil­ters. The image is ana­lyzed by a neur­al net­work, after which it is com­plete­ly redrawn.

On the left is the orig­i­nal pho­to, on the right is the result of Pris­ma. Source: periscope.com.ua

colorize

The ser­vice col­ors black and white pho­tos in real­is­tic col­ors.

This is a share­ware ser­vice. Each illus­tra­tion will have the inscrip­tion “Cre­at­ed by colorize.cc”, and you can upload up to 50 pic­tures in total. To remove the inscrip­tion and increase the down­load lim­it to 10,000, you need to pay $10.

An exam­ple of how Col­orize works.

RunwayML

An appli­ca­tion that com­bines many of the neur­al net­works that we have already talked about and adds oth­ers. Here you can gen­er­ate images, remove objects from pho­tos and videos, cre­ate text cap­tions for objects in pic­tures, cre­ate a 3D mod­el from pho­tos, and much more. All cal­cu­la­tions take place in the cloud. The appli­ca­tion has a web ver­sion, as well as ver­sions for Win­dows and Mac.

Run­way ML is free to use, but there will be restric­tions on upload­ing your work in high res­o­lu­tion and using some tools. The cheap­est plan with­out these restric­tions costs $35.

Conclusion

Time to learn tools that speed up pro­cess­ing and offer new cre­ative tech­niques. The main thing to remem­ber about the fea­tures of AI-based algo­rithms is that the result is not always com­plete­ly under the con­trol of the user. There­fore, clas­sic graph­ic edi­tors do not become a thing of the past, but work in syn­er­gy with new tools.