Real-Time Video Processing in .NET MAUI

🎥 Real-Time Video Processing in .NET MAUI

Applying Live Camera Filters with SKCanvas and SkiaSharp

Mobile applications are increasingly incorporating real-time computer vision, augmented reality, image enhancement, and live camera effects.

Whether you're building:

• 📸 A social media app
• 🎨 A photo editing tool
• 🤖 An AI-powered vision application
• 🛡️ A barcode or document scanner
• 🧠 A machine learning camera pipeline …you'll eventually need to process camera frames in real time.

Fortunately, combining .NET MAUI with SkiaSharp allows us to build powerful cross-platform video processing experiences using GPU-accelerated rendering and custom filters. In this guide, we'll build a real-time camera processing pipeline capable of:

• 🎥 Capturing live camera frames
• 🎨 Applying image filters
• ⚡ Rendering at interactive frame rates
• 🧠 Preparing frames for AI/ML inference
• 📱 Running across Android, iOS, Windows, and MacCatalyst

🧠 Understanding the Pipeline

Unlike static image processing:

Image
 ↓
Filter
 ↓
Render

Real-time processing becomes:

Camera
 ↓
Frame Capture
 ↓
Image Processing
 ↓
SKCanvas Rendering
 ↓
Display

This pipeline repeats 30–60 times per second.

⚡ Why Use SkiaSharp?

While MAUI graphics are excellent for standard UI rendering, video processing requires:

• High-frequency drawing
• Pixel manipulation
• Shader support
• Image effects
• GPU acceleration

This makes SkiaSharp an excellent choice.

📊 GraphicsView vs SKCanvas

For video processing, SKCanvas is the clear winner.

🏗️ Architecture Overview

A scalable design looks like:

Camera Service
      ↓
Frame Processor
      ↓
Filter Pipeline
      ↓
SKCanvasView
      ↓
Screen

This separation allows filters to evolve independently.

📦 Required Packages

Install:

dotnet add package SkiaSharp.Views.Maui.Controls

Optionally:

dotnet add package CommunityToolkit.Maui.Camera

or your preferred camera library.

📸 Capturing Camera Frames

Most camera libraries expose frames as:

byte[]

or

Stream

We convert them into:

SKBitmap

for processing.

🧩 Creating the Camera Service

public interface ICameraFrameProvider
{
    event EventHandler<SKBitmap> FrameReceived;
}

This abstraction keeps rendering independent from capture.

🎨 SKCanvasView Setup

XAML

<skia:SKCanvasView
    x:Name="CameraCanvas"
    PaintSurface="OnPaintSurface"/>

Backing Field

private SKBitmap? _currentFrame;

🔄 Receiving Frames

_camera.FrameReceived += (_, frame) =>
{
    _currentFrame = frame;

    MainThread.BeginInvokeOnMainThread(() =>
    {
        CameraCanvas.InvalidateSurface();
    });
};

Every new frame triggers a redraw.

🎥 Rendering Frames

private void OnPaintSurface(
    object sender,
    SKPaintSurfaceEventArgs e)
{
    var canvas = e.Surface.Canvas;

    canvas.Clear();

    if (_currentFrame is null)
        return;

    canvas.DrawBitmap(
        _currentFrame,
        e.Info.Rect);
}

At this point we have a live camera feed.

🎨 Applying Real-Time Filters

Now comes the fun part.

🌑 Grayscale Filter

private SKColorFilter CreateGrayScaleFilter()
{
    return SKColorFilter.CreateColorMatrix(new float[]
    {
        0.3f,0.3f,0.3f,0,0,
        0.59f,0.59f,0.59f,0,0,
        0.11f,0.11f,0.11f,0,0,
        0,0,0,1,0
    });
}

Apply During Rendering

using var paint = new SKPaint
{
    ColorFilter = CreateGrayScaleFilter()
};

canvas.DrawBitmap(
    _currentFrame,
    e.Info.Rect,
    paint);

Result: 📸 Live grayscale video.

🌈 Sepia Filter

private SKColorFilter CreateSepiaFilter()
{
    return SKColorFilter.CreateColorMatrix(new float[]
    {
        0.393f,0.769f,0.189f,0,0,
        0.349f,0.686f,0.168f,0,0,
        0.272f,0.534f,0.131f,0,0,
        0,0,0,1,0
    });
}

Classic vintage effect.

❄️ Cool Tone Filter

private SKColorFilter CreateCoolFilter()
{
    return SKColorFilter.CreateLighting(
        SKColors.LightBlue,
        SKColors.Black);
}

Useful for camera enhancement apps.

🔥 Dynamic Filter Switching

Create:

public enum FilterType
{
    None,
    GrayScale,
    Sepia,
    Cool
}

Current selection:

private FilterType _currentFilter;

Then:

private SKPaint CreatePaint()
{
    return _currentFilter switch
    {
        FilterType.GrayScale => new SKPaint
        {
            ColorFilter = CreateGrayScaleFilter()
        },

        FilterType.Sepia => new SKPaint
        {
            ColorFilter = CreateSepiaFilter()
        },

        _ => new SKPaint()
    };
}

Now filters can be changed at runtime.

* * *

⚡ Performance Optimization
==========================

Real-time rendering can become expensive quickly.

* * *

1️⃣ Reuse Bitmaps
-----------------

Bad:
```csharp
new SKBitmap(...)

every frame. Good:

bitmap.InstallPixels(...)

Reuse buffers whenever possible.

2️⃣ Limit Resolution

Processing:

1920x1080

for every frame may be unnecessary. Many apps use:

640x480

or

1280x720

for real-time pipelines.

3️⃣ Frame Skipping

Process:

Frame 1 ✔
Frame 2 ❌
Frame 3 ✔
Frame 4 ❌

This dramatically reduces CPU usage.

4️⃣ Background Processing

Never process frames on the UI thread.

await Task.Run(() =>
{
    ProcessFrame(bitmap);
});

🧠 Integrating AI Processing

One of the most powerful use cases is combining:

• Camera
• SkiaSharp
• TensorFlow Lite Pipeline:

Camera
 ↓
SKBitmap
 ↓
TensorFlow Lite
 ↓
Detection
 ↓
Overlay

Examples:

• Face detection 😀
• Object recognition 📦
• OCR 📄
• Barcode scanning 🏷️
• Pose estimation 🏃

🎯 Drawing Detection Overlays

Suppose an AI model returns:

public record Detection(
    string Label,
    SKRect Bounds);

Drawing becomes:

foreach(var detection in detections)
{
    canvas.DrawRect(
        detection.Bounds,
        overlayPaint);

    canvas.DrawText(
        detection.Label,
        detection.Bounds.Left,
        detection.Bounds.Top,
        textPaint);
}

This creates augmented-reality style overlays.

📊 Real-World Frame Rates

Actual performance depends on:

• Filter complexity
• AI workload
• Device GPU

⚠️ Common Mistakes

❌ Processing on UI Thread

Causes visible stuttering.

❌ Creating New Paint Objects Every Frame

Allocate once. Reuse often.

❌ Full HD Processing for Simple Effects

Most filters do not require maximum resolution.

❌ Forgetting Bitmap Disposal

bitmap.Dispose();

Memory leaks become obvious during long camera sessions.

🧩 Advanced Ideas

Once the pipeline exists, adding new features becomes easy:

🎨 Beauty Filters

• Skin smoothing
• Brightness enhancement

📷 QR Scanner

Real-time barcode recognition.

🧠 AI Classification

TensorFlow Lite integration.

🌎 AR Effects

Draw masks, stickers, or overlays.

🎥 Live Streaming Effects

Apply filters before streaming.

🔗 Reference Links

• https://learn.microsoft.com/dotnet/maui/
• https://github.com/mono/SkiaSharp
• https://learn.microsoft.com/dotnet/communitytoolkit/maui/

🎬 Final Thoughts

Real-time video processing opens the door to an entirely new class of mobile experiences. By combining .NET MAUI with SkiaSharp, you can build applications that don't just display camera feeds—they actively understand, transform, and enhance them.

Whether you're creating an AI-powered scanner, a social media camera, or an augmented reality experience, the combination of MAUI and SKCanvas provides a powerful and flexible foundation for modern computer vision applications. 🎥🚀