Date Revision Number Author Revision Content 27/11/2010 0.6.3 Brad Hards Fix tracker link and a couple of typos. 03/03/09 0.6.2 Julien Kerihuel Add configuration info for server mode. 01/02/09 0.6.1 Julien Kerihuel Add configuration info for server mode. 04/01/09 0.6 Julien Kerihuel server mode documented, update mapiproxy naming to MAPIProxy. 29/12/08 0.5.5 Julien Kerihuel Add 3 new questions to FAQ section 09/12/08 0.5.4 Julien Kerihuel Add dcesrv:assoc group checking to smb.conf configuration requirements 10/07/08 0.5.3 Julien Kerihuel Rename smbd process to samba session API and update documentation 08/26/08 0.5.2 Julien Kerihuel documentation update on NSPI replacement and new FAQ question added 08/26/08 0.5.1 Julien Kerihuel documentation on NSPI referral added 08/11/08 0.5 Julien Kerihuel unbind hook added, cache module documentation and scenario added 07/23/08 0.4 Julien Kerihuel MAPIProxy API hooks, IDL update, mapiproxy structure description and documentation added for the cache module 06/25/08 0.3.2 Julien Kerihuel Minor installation update 06/04/08 0.3.1 Brad Hards Minor edits 05/27/08 0.3 Julien Kerihuel Available modules section added 05/24/08 0.2 Julien Kerihuel EMSMDB protocol version subsection updated, modules system section added, 5-minute configuration updated 05/15/08 0.1 Julien Kerihuel Initial Revision
MAPIProxy is an endpoint server for Samba4 which proxies ExchangeRPC traffic from MAPI clients (Outlook, openchangeclient, etc.) to Microsoft Exchange Server (and back). It can either act as a transparent proxy, for hacking, monitoring or debugging purposes or modify traffic on the fly and so provide new features. It is primarily developed for - but not limited to - third-party implementors looking for a development framework they can use for MAPI acceleration purposes.
This project is originally based on dcerpc_remote.c code from Stefan Metzemacher (Samba4 trunk) and is released under GPLv3 or later. It creates a dynamic shared object file which is loaded into samba and uses the Samba configuration file (smb.conf) to set common options.
Figure 1. General MAPIProxy network overview
The MAPIProxy traffic can be divided into 3 different parts as described in the figure above:
If you find bugs, limitations or have features you would like to see included in MAPIProxy, please register on the OpenChange Tracker System and create new tickets.
MAPIProxy is only available through SVN at the moment. A tarball release will only be made when we have a stabilized API with a preliminary set of useful features. You will need a SVN client to download openchange (including MAPIProxy).
$ svn co https://svn.openchange.org/openchange/trunk openchange
The MAPIProxy implementation requires a very recent Samba4 version in order to run properly. If Samba4 is planned to be installed from scratch for MAPIProxy only, please use the make samba-git compilation rule provided in the build system. This command will automate most part of the samba4 installation process. The only requirement for this step is to have an up to date GIT version installed on the system.
# make samba-git
When the installation process is finished, a running samba4 installation will be located in /usr/local/samba/. You will possibly be required to run ldconfig before you move to next steps. Please refer to doc/howto.txt for further information on openchange compilation.
If you have existing OpenChange DSO in the /usr/local/samba/modules/dcerpc_server/ folder, such as dcesrv_exchange.so, please remove them prior loading samba with MAPIProxy.
$ ./autogen.sh $ ./configure --prefix=/usr/local/samba $ make # make install # rm -rf /usr/local/samba/modules/dcerpc_server/dcesrv_exchange.so
This 5-Minute configuration will help you set up a minimal MAPIProxy using specified credentials and relaying traffic from Outlook clients to a remote Exchange server. This configuration will be performed in three steps:
# ./setup/provision --realm=OPENCHANGE.LOCAL --domain=OPENCHANGE --adminpass=openchange --server-role='domain controller'
If you don't have DNS resolution and your realm can't be resolved, samba will be unable to authenticate the user in its user database. You must specify a realm which MAPI clients and MAPIProxy can resolve.
If everything works fine, the provisioning script will have created all the databases, populated the AD (Active Directory) and generated a valid smb.conf file.
In this configuration, we'll set the same credentials both for the user in the windows domain and on the Samba4 server. Let say there is already a user named testuser with its password set to openchange on the Exchange server:
# ./setup/newuser testuser New Password: openchange
In this final step, we only need to customize a small set of parameters:
[globals]
netbios name = MAPIPROXY
workgroup = OPENCHANGE
realm = OPENCHANGE.LOCAL
server role = domain controller
### Configuration required by mapiproxy ###
dcesrv:assoc group checking = false
dcerpc endpoint servers = epmapper, mapiproxy
dcerpc_mapiproxy:binding = ncacn_ip_tcp:192.168.1.1[print]
dcerpc_mapiproxy:username = testuser
dcerpc_mapiproxy:password = openchange
dcerpc_mapiproxy:domain = EXCHANGE
dcerpc_mapiproxy:interfaces = exchange_emsmdb, exchange_nsp, exchange_ds_rfr
dcerpc_mapiproxy:modules = downgrade
### Configuration required by mapiproxy ###
[netlogon]
path = /usr/local/samba/var/locks/sysvol/openchange.local/scripts
read only = no
[sysvol]
path = /usr/local/samba/var/locks/sysvol
read only = no
We are now ready to run samba:
# samba -d5 -i -M single
If everything works properly, the following lines should be displayed in samba output:
DCERPC endpoint server 'exchange_emsmdb' registered DCERPC endpoint server 'exchange_nsp' registered DCERPC endpoint server 'exchange_ds_rfr' registered DCERPC endpoint server 'mapiproxy' registered dcesrv_interface_register: interface 'epmapper' registered on endpoint 'ncacn_np:[
ipe\pmapper]' dcesrv_interface_register: interface 'epmapper' registered on endpoint 'ncacn_ip_tcp:[135]' dcesrv_interface_register: interface 'epmapper' registered on endpoint 'ncalrpc:[EPMAPPER]' MAPIPROXY module 'downgrade' registered MAPIPROXY module 'downgrade' loaded mapiproxy_module_load 'downgrade' (Downgrade EMSMDB protocol version EcDoConnect/EcDoRpc) dcesrv_interface_register: interface 'exchange_emsmdb' registered on endpoint 'ncacn_np:[
ipe
s]' dcesrv_interface_register: interface 'exchange_emsmdb' registered on endpoint 'ncacn_np:[
ipe
rotected_storage]' dcesrv_interface_register: interface 'exchange_emsmdb' registered on endpoint 'ncacn_ip_tcp:' dcesrv_interface_register: interface 'exchange_nsp' registered on endpoint 'ncacn_np:[
ipe
s]' dcesrv_interface_register: interface 'exchange_nsp' registered on endpoint 'ncacn_np:[
ipe
rotected_storage]' dcesrv_interface_register: interface 'exchange_nsp' registered on endpoint 'ncacn_ip_tcp:[]' dcesrv_interface_register: interface 'exchange_ds_rfr' registered on endpoint 'ncacn_np:[
ipe
s]' dcesrv_interface_register: interface 'exchange_ds_rfr' registered on endpoint 'ncacn_np:[
ipe
rotected_storage]' dcesrv_interface_register: interface 'exchange_ds_rfr' registered on endpoint 'ncacn_ip_tcp:[]'
You should now be able to configure Outlook to use an Exchange account with the proxy IP address and run Outlook seamlessly (both online or cached exchange mode).
When Outlook sets up an Exchange account using either the mail applet from the configuration panel or the account editor within Outlook, it uses the NSPI protocol (Name Service Provider Interface, effectively the address book provider). In this case, NSPI is used to resolve the Exchange username and fetch from Exchange server all information needed by Outlook to initiate direct connection to the EMSMDB pipe (effectively the message store) the next time it connects to the server.
At some point of the profile's creation process, Outlook queries Exchange for some specific connection information using the NspiGetProps (0x9) RPC operation . More specifically, when Outlook requests for the PR_EMS_AB_NETWORK_ADDRESS MAPI property, Exchange returns a list binding strings. Outlook next stores these binding strings at some location - associated to the Outlook profile - in the windows registry and uses them for future connections.
Outlook can also rely on other information returned by NSPI functions and connect to the real Exchange server rather than MAPIProxy. Such case occurs when Outlook is able to resolve the exchange server using its hostname. This reference to the original Exchange server can be found when Outlook requests for the PR_EMS_AB_HOME_MDB MAPI property during the NspiQueryRows (0x3) RPC operation. MAPIProxy replaces the Exchange server name with its own netbios name and forward the reply to the client.
In the meantime, this information is next used by Outlook to query a minimal entry ID for a distinguished name using this server name. MAPIProxy needs to substitute the server name in the inbound request string with the original exchange one.
MAPIProxy needs to avoid Outlook clients being aware of this remote server address and trying to communicate directly with the remote server instead of using the proxy. In order to do this, MAPIProxy alters the Outlook-Exchange MAPI traffic and replaces these binding strings with the MAPIProxy FQDN and netbios name.
The Address Book Name Service Provider Interface (NSPI) Referral Service is a service used by Outlook to retrieve the name of an NSPI server. No NSPI connection should be initiated without first querying for the correct NSPI server. In this case, RFR returns the fully qualified domain name of the real Exchange server and starts using it if available.
MAPIProxy needs to avoid Outlook clients being aware of this server address and trying to communicate directly with the remote server instead of using the proxy. In order to do this, MAPIProxy alters the Outlook-Exchange MAPI traffic and replaces the server DN returned by RfrGetNewDSA (0x0) RPC operation with the MAPIProxy realm as specified in smb.conf.
When Outlook starts and presumably calls MapiLogonEx, it first opens a connection to the Exchange server on the NSPI pipe, then on the EMSMDB pipe. Under Outlook 2003, the very first EMSMDB RPC call Outlook makes can be considered as a kind of protocol version negotiation. Depending on which version of Outlook is used, and how the Exchange server replies to the EMSMDB connect request, Outlook will either keep using the same pool of RPC calls or downgrade.
For example Outlook 2003 (default behavior) tests if the remote server supports the 2 new EMSMDB calls (EcDoConnectEx/EcDoRpcExt2) introduced in Exchange 2003. If Exchange replies to the EcDoConnectEx request with a dcerpc_fault, it means the server does not support the RPC operation, presumably has a version before 2003, and Outlook needs to downgrade its version in order to communicate with the server:
If MAPIProxy runs in an environment with Outlook clients and Exchange servers using a version above 2003, a last step is required to successfully use Outlook. The EcDoConnect RPC reply returns the Exchange server version (as an array of 3 short integers). When Outlook detects this particular server version, it automatically closes the connection and keep requesting indefinitely for EcDoConnectEx. To deal with this, MAPIProxy modifies the EcDoConnect reply sent by Exchange and replaces the server version with a one equal to that sent by Exchange 2000.
In the meantime, if we reproduce this test with Outlook 2000 which doesn't support these 2 new RPC calls, Outlook will directly call EcDoConnect.
The main difference between the EcDoConnectEx/EcDoRpcExt2 operations and the EcDoConnect/EcDoRpc operations is that the former use both XOR 0xA5 obfuscation and LZ77 compression/Direct2 encoding; while the latter only use the XOR obfuscation to handle MAPI content. If MAPIProxy wants to act as an intelligent proxy (for example, to be able to analyze MAPI content on the fly, compress MAPI data etc), receiving non compressed MAPI traffic would probably improve the overall process.
Below is a list of Exchange/Outlook pairs and the EMSMDB connect function they will use by default: Exchange version Outlook version EMSMDB connect function 5.5/2000 any EcDoConnect (0x0) 2003 2000 EcDoConnect (0x0) 2007 2000 EcDoConnect (0x0)
Microsoft officially says it is unsupported 2003 2003-2007 EcDoConnectEx (0xa) 2007 2003 EcDoConnectEx (0xa) 2007 2007 EcDoConnectEx (0xa)
MAPIProxy reproduces the Exchange 2000 behavior and prevents Outlook from communicating with the Exchange server using the EcDoConnectEx/EcDoRpcExt2 as described in Figure 2 below. When Outlook sends an EcDoConnectEx request, MAPIProxy does not relay the request to the remote Exchange server and immediately returns a dcerpc_fault to Outlook. Outlook, assuming the server doesn't support this call uses EcDoConnect instead. From this call, MAPIProxy relay the information to Exchange.
Figure 2. MAPIProxy behavior on Outlook EMSMDB connection
From the Exchange side, the server will analyze this EcDoConnect request as a call sent by Outlook 2000 or below version. Exchange works fine using this protocol version unless Exchange 2007 SP1 which appears to introduce client version restrictions by default. In the meantime, existing tests demonstrate similar restrictions would apply to Outlook 2003 connection (without MAPIProxy) and prevent Outlook version before 2007 connecting to Exchange 2007. Further information and solution is available at the following addresses:
IDL stands for Interface Definition Language and OpenChange uses this format to describe ExchangeRPC communications. This file is processed by pidl (Perl IDL compiler provided by Samba4) which turns this protocol description into C-code dealing with the push, pull and print operations.
OpenChange development policy in trunk used to push a new MAPI call in the IDL only when the associated libmapi implementation and mapitest unit is developed, but this was preventing from distributing MAPIProxy with further openchange releases. Furthermore, the OpenChange IDL is now almost complete and merging back to the trunk helps improving libmapi reliability.
The MAPIProxy stackable modules system provides implementers a development framework to add new features. This stackable mechanism allows developers to write modules with a very specific scope of which modifications will transparently be relayed to the next module until it is finally pushed by MAPIProxy to the next hop (Figure 3.).
Figure 3. MAPIProxy module stack and EcDoRpc interaction
With this system, developers can focus their effort on ExchangeRPC traffic - or any other protocol samba supports - interception, modification, analysis and avoid spending time on implementing a new endpoint server. Furthermore it provides an easier way for implementers to divide the work in smaller units and develop each of them in a separated module.
MAPIProxy modules are dynamic shared objects with an entry point and a limited set of hooks. These modules have to be installed in the dcerpc_mapiproxy folder within the samba4 modules directory (e.g. /usr/local/samba/modules). MAPIProxy modules specified in the Samba configuration file (smb.conf) will be loaded into MAPIProxy at runtime and interact with each other in the same order they were defined:
dcerpc_mapiproxy:modules = downgrade,dummy
All MAPIProxy modules will be registered but only those specified on the dcerpc_mapiproxy:modules parametric option line will be added to the chained list of effective modules.
MAPIProxy modules must have an entry point function named samba_init_module. This function needs to set general information about the module, specify the module's hooks and finally call the mapiproxy_module_register function to register itself in the MAPIProxy module subsystem.
NTSTATUS samba_init_module(void)
{
struct mapiproxy_module module;
NTSTATUS ret;
/* Fill in our name */
module.name = 'sample';
module.description = 'A sample module';
module.endpoint = 'any';
/* Fill in all the operations */
module.init = sample_init;
module.push = sample_push;
module.ndr_pull = sample_ndr_pull;
module.pull = sample_pull;
module.dispatch = NULL;
module.unbind = NULL;
/* Register ourselves with the MAPIPROXY subsytem */
ret = mapiproxy_module_register(&module);
if (!NT_STATUS_IS_OK(ret)) {
DEBUG(0, ('Failed to register 'sample' mapiproxy module!));
return ret;
}
return ret;
}
In the meantime, it can happen that a module requires to interact with more than a single interface. In such case, use the 'any' keyword which will call the modules functions with any endpoints proxied by MAPIProxy.
MAPIProxy offers a set of hooks which modules can implement to modify/change/alter client to server MAPI traffic. The figure below shows how and when hooks are called during a request/response lifetime.
Figure 4. Usage of MAPIProxy Hooks during a request/response life time
Please note that the module API is still under development and is likely to change in further revisions.
MAPIProxy uses a structure modules can modify in their dispatch routine and which impact on the general MAPIProxy behavior.
Figure 5. overview of mapiproxy structure variables scope
The downgrade module implements the EcDoConnect/EcDoRpc negotiation as described in section 4.2. It ensures Outlook will not send compressed information or use functions other than EcDoRpc for EMSMDB transport. In order to use the downgrade module, edit smb.conf and add downgrade to dcerpc_mapiproxy:modules.
dcerpc_mapiproxy:modules = downgrade
Note that this module only works with an infrastructure using two or more instances of MAPIProxy as described in Figure 1
The pack module implements routines designed to manipulate and factorize MAPI content between different MAPIProxy instances. It also offers a developer overview on how to manipulate mapi requests. Last but not least, it provides data which can next be used by subsequent MAPIProxy modules for example to compress or encrypt this proxypack blob.
Figure 6. Pack process
Figure 7. Unpack process
This module has two configuration options:
mpm_pack:opnums = 0x70,0x75,0x76,0x77,0xa
mpm_pack:lasthop = true
In order to use the pack module, edit smb.conf and add pack to dcerpc_mapiproxy:modules.
dcerpc_mapiproxy:modules = downgrade,pack
The cache module implements a cache mechanism for streams related to messages or attachments. This module reduces communication latency between MAPI clients (using online mode) and Exchange. When configured with online mode, MAPI clients retrieve data from Exchange each time they access a message and don't have any offline storage mechanisms enabled - data are downloaded and stored within a temporary files folder. This module also offers a preliminary synchronization mechanism which can be used to transfer files between different MAPIProxy instances and use different protocols than MAPI for data transfer (such as rsync or wget).
The cache module is designed to cover different cases:
This scenario only requires a single MAPIProxy instance and requires a single configuration option:
mpm_cache:path = /tmp/cache
Figure 8. Replay stream scenario
This scenario requires two MAPIProxy instances and requires different configuration options for local and remote MAPIProxy:
mpm_cache:path = /tmp/cache
mpm_cache:ahead = false
mpm_cache:sync = true
mpm_cache:sync_cmd = /usr/bin/rsync -z mapiproxy@192.168.102.2:__FILE__ __FILE__
mpm_cache:path = /tmp/cache
mpm_cache:ahead = true
mpm_cache:sync = false
Figure 9. Read ahead scenario with synchronization mechanism
The module monitors OpenMessage, OpenAttach, OpenStream, ReadStream and Release MAPI calls and stores streams on the local filesystem with indexation in a TDB database. Note that the module doesn't yet provide semantics needed to remove entries from the TDB database.
This module has different configuration options and modes:
mpm_cache:path = /tmp/cache
mpm_cache:ahead = true
mpm_cache:sync = true
mpm_cache:sync_cmd = /usr/bin/rsync -z mapiproxy@192.168.102.2:__FILE__ __FILE__
In order to use the cache module, edit smb.conf and add cache to dcerpc_mapiproxy:modules.
dcerpc_mapiproxy:modules = downgrade,cache
This 5-Minute configuration will help you set up a preliminary OpenChange server. This configuration will be performed in three steps. Before running these commands, make sure you have followed step 1 (Provision Samba) and step 2 (Add a user account) in MAPIProxy 5-Minute configuration section.
# ./setup/openchange_provision
This script will extends Samba4 Active Directory with Exchange classes and attributes needed to run OpenChange server. Note that this operation may require several minutes to complete.
# ./setup/openchange_newuser --create <username>
where username is the user account you want to give access to OpenChange server
# ./setup/openchange_provision --openchangedb
# ./setup/openchange_newuser --mailbox <username>
[globals]
netbios name = MAPIPROXY
workgroup = OPENCHANGE
realm = OPENCHANGE.LOCAL
server role = domain controller
### Configuration required by OpenChange server ###
dcerpc endpoint servers = epmapper, mapiproxy
dcerpc_mapiproxy:server = true
dcerpc_mapiproxy:interfaces = exchange_emsmdb, exchange_nsp, exchange_ds_rfr
### Configuration required by OpenChange server ###
[netlogon]
path = /usr/local/samba/var/locks/sysvol/openchange.local/scripts
read only = no
[sysvol]
path = /usr/local/samba/var/locks/sysvol
read only = no
Although section 1.1 only describes MAPIProxy as a proxy, recent work makes it possible to turn MAPIProxy either into a complete and real stand-alone server or server/proxy hybrid.
MAPIProxy behaviour is controlled through the dcerpc_mapiproxy:server parametric option. To use MAPIProxy as an independent server, set
dcerpc_mapiproxy:server = true
In addition to the server mode described above, MAPIProxy provides an additional set of configuration options which makes possible to override and customize MAPIProxy behavior. The server mode has been designed to supply a modular mechanism somewhat similar to the modules one described in section 5. While MAPIProxy modules are stackable and can be chained, server modules only support a single module for a given endpoint:
Figure 10. Server mode enabled
However there may be some cases where developers would like to run a custom server they have developed, or handle a limited set of ExchangeRPC traffic on their own for a given endpoint. This configuration is made possible through 3 parametric options:
dcerpc_mapiproxy:nspi_server = nspi_server
dcerpc_mapiproxy:emsmdb_server = emsmdb_server
dcerpc_mapiproxy:rfr_server = exchange_ds_rfr
Each of these options specifies the server module name to be loaded for a given endpoint. Note that these options override the dcerpc_mapiproxy:server state:
Figure 11. Server mode enabled but custom NSPI server loaded
Figure 12. Server mode disabled but NSPI server loaded
Figure 13. Outlook error: The action could not be completed
If you have followed the 5-Minute Configuration instructions and the above error message box (Figure 13) is displayed each time you click the Check Name button, then you need to:
Figure 14. Resolution: Always prompt for username and password
Next time you click on Check Name, Outlook will prompt for username and password. A similar credentials dialog will be displayed each time Outlook is launched.
The profile was properly created using the mail applet from the configuration panel (or using Outlook wizard). However when I launch Outlook, I keep having the following error message:
Figure 15. Outlook error: Unable to Open your default e-mail folders
This probably means Outlook is unable to lookup the resolved name of your MAPIProxy/samba4 server. You can either:
C:WINDOWSystem32\tcriversr mapiproxy. For example if I have mapiproxy.openchange.local pointing at 192.168.102.2, then hosts file should hold the following line: 192.168.102.2 mapiproxy.openchange.local mapiproxy
No it doesn't. MAPIProxy works fine as a member server of a Windows domain. However, since delegated credentials and forwarded kerberos credentials don't yet work, you'll need to force samba to rely on the local SAM database. To force this behavior, add to smb.conf within the global section:
server role = member server
aux_methods:member server = sam
For some configuration, the private keys generation process at Samba startup can be very long. In case private keys are not generated within a couple of minutes, it is suggested to recompile Samba with gnutls disabled as in the example below:
$ ./configure.developer --enable-debug --disable-gnutls
$ gmake idl_full
$ gmake
$ sudo gmake install
On Ubuntu, I have the following output while trying to install samba4 from OpenChange sources:
To build Samba, run /usr/bin/make
Step2: Compile Samba4 (IDL)
./script/installsamba4.sh: 332: gmake: not found
Step3: Compile Samba4 (Source)
./script/installsamba4.sh: 332: gmake: not found
Error in Step3 (error code 127)
gmake is make on Ubuntu. Creating the following symbolic link will fix the issue:
$ sudo ln -s /usr/bin/make /usr/bin/gmake