Overlays

You put the overlay on a separate artboard and when you link to it, click on Overlay at the top of the popover. It automatically adds a dark background which you can change the opacity of. Setting the opacity to 0 will hide it. Checking "Is a Toggle" will cause the trigger layer to open and close the overlay as a toggle. If you want a separate layer to close the overlay, create a link and select the Close button that appears at the top. You can link overlays to other overlays. If you want the overlay to replace the previous one rather than stacking on top of it, check "Replace previous overlay". This option is only available when linking from an overlay. Overlays can be shared by multiple artboards.

When creating static screens it's common to show the screen below the overlay. You can do this by turning the artboard that is behind the overlay into a symbol. Then in the overlay artboard you would place a symbol instance on it and cover it with a transparent rectangle so it dims. In the prototype it won't show the reference symbol instance and the "dimming" layer.

Animated Layers

Alternatively you can use an animated layer to be an overlay though it can't be shared by multiple artboards. Add your overlay to the same artboard that it will appear on, then create a link or hotspot and press on the Animation button at the top. Since the overlay is visible you would select "End State" for "Current design is". In the animation editor move the overlay to the hidden state. You can use the Off-Artboard button to have it just on the edge of the artboard. To have a close button close the overlay, create a link within the animation editor that links to the start state. Outside the animation editor, you can hide the overlay layer so you can work on the rest of the design and it will still appear in the prototype.