I'm gonna fall over soon, so see if this makes sense to you. If you're working with Photoshop to meld 2 separate pieces-
1) Lay the background layer on top of the render, and set the opacity to "Multiply".
2) Make a selection of the render portion below (I presume it has no background or one that can easily be removed from the selection), and then invert your selection.
3) Under the Selection Menu in the top MenuBar, expand your selection 1 pixel and then add a 1 pixel feather to your selection.
4) Use that selection to create a mask in the menubar layer menu, set to "Reveal Selection"
5) Use the brush tools to edit your mask (have to be clicked on the mask portion in the Layers palette preview to make affect the mask rather than the image, cleaning up edges where they don't quite work, or where you'd like to layer the feathering, by changing the softness of the brush tool as you work in that area. Painting with black will add to the mask, painting with white will remove from it (I might have that backwards, but I believe that's the direction)
6) Reset your top layer with the background to Normal Opacity, and do a check of your edges again, cleaning up the mask more if you need to.
While it seems backwards to have the background layer on top while you do this, if it's the smaller area, it tends to be easier, and in the final image, there's no difference, because it's all about how you've set the edges of the mask to blend.
If you want to create a drop shadow on the rendered portion:
1) Select the render again, and create a channel this time. This one is going to be a little bit of dark edge on the bottom of the render, so see how soft you want the edges to be. You can use the blur filter to get it to be exactly where you want it to be, again, making sure that you're selected on the channel in the channel palette.
2) Then you're going to make one more channel, starting by selecting the render again, and creating a new channel. This channel you're going to pick up and move in the direction where you want your shadow to fall on the background area. Use the blur filter again to set the softness of the edge.
3) Click on one of the channels, and there are various ways to do it, but the easiest to explain is - use the flyout menu on the palette to "Make a Selection"
4) Click on the other channel, and use the flyout menu, this time to "Intersect with selection".
5) Go back to your layers palette, and make a new layer on top of the previous 2 layers (the mask should appear automatically this time).
6) Select the layer preview in the palette, and fill the whole layer with color - black if that's what you want, something else if you think it will blend better.
7) Set the layer opacity to "Multiply", and fiddle with the transparency until it's about as dark as you want it to be.
8) Using your brushes again, clean up anything you want to on the mask, adjusting the size and softness of your brush to give you the effect you want.
Pardon me, is it that you want them to seem more intertwined, or that you want it not to look like the render is literally sitting on top of the image, rather than coming across it?
1) The drop shadow is working against you there in that. If you soften it up more, it might keep some of the distinction you want, but also allow it to blend more.
2) Copy the background on top of of the curly render part, and set the Opacity Type to Multiply, and then set the transparency level to somewhere between 5% and 10%, fiddle around a bit until you get it closer to what you want. You can blur that laid directly on top of the curly part, or you can mask out the curly part and set your edges to do a wide feather so that the background "fades in" on the edges of the render.