hackndo

Spray passwords, avoid lockouts

Tue, 04 Jun 2024 08:25:55 +0000

Password spraying is a well-known technique which consists of testing the same password on several accounts, in the hope that it will work for one of them. This technique is used in many different contexts: On web applications, the Cloud, services like SSH, FTP, and many others. It’s also widely used in internal penetration testing with Active Directory. It’s the latter that we’re going to focus on, because although the technique seems simple, it’s not easy to put it into practice without side effects.

Introduction

This article is not about something new, but rather a report on my research into password policies in an Active Directory environment. Indeed, there are several ways of limiting an attacker by locking accounts. These different levers are very useful when they are understood, which is not always the case (and wasn’t the case for me a few weeks ago). This article will, I hope, clarify what password policies allow, how they are applied, and therefore, as a pentester, how to do password spraying while minimizing the risk of locking accounts.

Authentication mechanism

Password spraying on an Active Directory directory consists in choosing a password considered highly likely to be used by at least one user, and testing it on all users in the domain. We won’t go into detail here on how to test the validity of a password, as there are many different ways of doing this. A password can be tested via Kerberos or NTLM, via different services (SMB, LDAP, etc.). In any case, when authentication is tested, the domain controller will first check whether the account is locked by the password policy. If not, it will check the validity of the password. Finally, if the password is valid, the account will be authenticated, and the user’s LDAP badPwdCount attribute will be reset to 0. If, on the other hand, the password is incorrect, then badPwdCount will be incremented, and if this failure results in the account being locked, according to password policy criteria, then the account will be marked as locked. This is done by ptoviding the current date and time in the user’s lockoutTime LDAP attribute.

That’s quite a verification process, which I’ve tried to summarize in a diagram. Does this make things clearer? I’m not sure. But here it is anyway.

Now that we’ve clarified this logic, we understand that there’s one unknown factor that seems rather important: the password policy. Because it’s by following the password policy applied to the account that the domain controller is able to know whether or not the account is locked, or whether it should be. Let’s take a look at where this default policy can be found, and explain the parameters we’re interested in.

The default password policy

When an Active Directory is installed, many things are created and set up by default. Among them are two GPOs: The Default Domain Policy linked to the domain root, and the Default Domain Controller Policy linked to the Domain Controllers OU.

The Default Domain Policy contains the following parameters for defining account lockout rules.

Account lockout duration : When an account is locked out, this parameter defines the time during which the account remains locked.
Account lockout threshold: Determines the number of incorrect attempts allowed. If this parameter is 3, then 3 incorrect attempts will lock the account.
Reset account lockout counter after: As a general rule, when an authentication fails, badPwdCount is incremented. However, if the previous failed attempt is older than the time set here, then badPwdCount is reset to 0. For example, if this parameter is set to 5 minutes, then on a first failure, badPwdCount is set to 1. On a 2nd failure a few seconds later, badPwdCount is set to 2. If a 3rd failure occurs, but 5min10 after the 2nd failure, badPwdCount will have been reset to 0, and this failure therefore sets it back to 1.

These are the parameters that the domain controller takes into account to determine whether an account is locked out. In other terms (and because the previous diagram wasn’t complex enough, of course), these values are used in the following way:

As you can see, the bad password counter is calculated by taking the date of the last logon failure, and if the time defined in Reset account lockout counter after has passed, then the counter is reset to zero. Otherwise, it is incremented. This counter is then compared with Account lockout threshold. If it’s equal to (or greater than) this threshold, then the account is locked out, and the lockoutTime field is filled in. On a subsequent attempt, if the time defined in Account lockout duration has passed (using the date in lockoutTime as a starting point) then the account is no longer locked, and the password will be tested.

If it’s still not clear, then take these explanations from the beginning, and concentrate. I can’t explain it any better. All this has to be clear to understand what’s coming next. We’ve already talked about the password policy in the Default Domain Policy GPO. But we’ll see that this is not the only place where this policy can be defined.

GPOs application order

Personally, I like to create a new, dedicated GPO for every settings (or group of settings) I want to apply to my users or computers. To apply a password policy, I’ll create a dedicated GPO, in which there will only be settings related to the password policy. And it’ll work just fine. Indeed, we’ve seen that the password policy can be defined in the Default Domain Policy, but they can obviously be set in any GPO.

But how does it work when several GPOs contain different password policies?

There is in fact a priority order which defines which GPO takes precedence in the event of a conflict. In the Group Policy Management Console, when you click on an OU (or on the domain), you’ll see the list of applied GPOs in the Linked Group Policy Objects tab. You’ll notice that they are numbered, via the Link Order column, which makes it possible to determine the priority of each GPO. Specifically, the last GPO in the list will be applied first, up to GPO number 1. This means that GPO number 1 has the “last word”. If its settings conflict with those of other GPOs, its own settings will effectively take effect.

In this example, if one password policy is defined in PASSWORD POLICY and another in Default Domain Policy, the latter will be applied.

There is no attribute that allows you to read the GPO number in the list. For each Organizational Unit (and for the domain object), the gpLink attribute contains the list of GPOs that are linked to this object, and they are saved in order of application, i.e. in decreasing order of Link Order. On the hackn.lab object, the PASSWORD POLICY GPO is listed first in its gpLink attribute, then LOCKSCREEN, FIREWALL and finally Default Domain Policy.

So… By listing the GPOs that are applied to the domain object, we’re now able to find out which password policy is actually being applied. Of course, it doesn’t stop there. What if we want to apply a different, stronger password policy to the administrators, for example?

Password policy on an Organizational Unit

Before even starting this chapter, I’m announcing that this won’t work. We’ll have to use PSOs to achieve this goal, which we’ll see right after.

We might be tempted to organize our Active Directory in such a way that our administrators are in a dedicated Organizational Unit, and we’d like to apply a specific password policy to these users. Why not create a GPO dedicated to this OU, in which we define our robust criteria for secure passwords?

Let’s take this example. We have a PASSWORD POLICY GPO placed at the domain level, which allows you to make 10 mistakes before locking accounts for 10 minutes.

Let’s then create a much more robust GPO linked to the Admins OU in which the adm_pixis account resides. This GPO should lockout an account for 60 minutes after 2 failed attempts.

Now we can try to log in with the user adm_pixis, but we deliberately get the password wrong. Once, twice, … three times, four times. The account is still unlocked.

On the 11th try, however, it doesn’t fail: our account is locked.

Why this black magic? Well, simply because when defining a password policy in a GPO, it will update some specific LDAP attributes of the object to which they are linked, and these attributes only exist on the domain object, not on the Organizational Units.

Here you have the lockoutDuration attribute, which corresponds to the Account lockout duration parameter, lockoutObservationWindow set by Reset account lockout counter after, or lockoutThreshold which can be defined with Account lockout threshold. So, when a password policy is defined by GPO and linked to an OU, this policy will have no effect on this Organizational Unit, and therefore no effect on domain accounts in this OU.

If you apply a GPO with a password policy to an OU that contains computers (workstations or servers), it has no effect on the domain policy, but it will apply to the local accounts of said computers.

All this means that it seems only possible to apply a global password policy to all users. But how do we harden this policy for our administrators? This is where PSOs, or Password Settings Objects, come in.

PSOs

Ah, PSOs. The answer to all our problems. PSOs are Active Directory objects that let you define the same password policy settings as those found in GPO, and apply them to users or groups of users. This makes it possible to create Fine-Grained Password Policies (FGPP).

These objects must be created in a very specific place, in a container called Password Settings Container, itself in the System container at the root level. A simple way to create a PSO is to use Active Directory Administrative Center tool and navigate to this container.

From this console, you can easily create password policies and decide to whom these policies should apply. If we wanted to take over our super secure policy, we could create a PSO with these same parameters and apply it to all members of the Domain Admins group.

The same question of priority arises as for GPOs. Firstly, PSOs have priority over the domain’s password policy. But then, if several PSOs are created, and users are affected by different PSOs, which one will be taken into account?

It’s the Precedence parameter present in the PSO, just after its name, that is used to sort PSOs. Like the Link Order for GPOs, PSOs are applied from highest to lowest Precedence. This means that the lowest values have priority over the highest, since they also have the last word.

If two PSOs ever have the same value in Precedence, the last PSO created will take precedence.

Let’s get a little more technical. We’ve seen that when GPOs set the password policy, an order of priority will be followed and the attributes of the domain object will be updated to reflect the actual password policy. These domain attributes are readable by all authenticated users, so all you have to do is read the attributes on the domain object to find out the default password policy applied to the domain.

What about PSOs? How do you know if a user is affected by a particular PSO? PSOs can’t modify domain attributes, since by their very nature they are designed to have different password policies. In fact, as each PSO is an Active Directory object, password policies are stored in the attributes of the object.

The attributes don’t have exactly the same names as on the domain, which would be too simple, but they’re still there. Still with password spraying in mind, we’re particularly interested in msDS-LockoutThreshold and msDS-LockoutObservationWindow.

Another attribute of great interest to us is msDS-PSOAppliesTo. It contains the list of users and/or groups to which the PSO applies.

PSO container rights problem

At this stage, you might think that we have all the information we need to know the effective password policy for each user. By default, we take the domain password policy by reading the domain object attributes, and then we list all the PSOs in the right order of priority, list the users in the groups on which each PSO is applied, and we can thus deduce which PSO is applied to which user.

In theory, this would work, but there’s a small hitch in this approach.

Can you see the problem? No ? By default, only administrators have the right to list PSOs. So, as an ordinary user, we have no way of listing the contents of the Password Settings Container, and therefore of seeing the PSOs, the policies applied, and to whom they are applied. For our purposes of password spraying, this is quite dangerous. This means we can never be certain of knowing the effective password policies for various domain users, as there is always a risk that a PSO (which we cannot see or read) is applied to one or more users.

Constructed Attributes & Backlinks

It was while facing this wall that I discovered constructed attributes. You probably know that when a user is added to a group, the member attribute of the group is updated, adding the user. However, if you simply list a user’s attributes, you won’t see any LDAP memberOf attribute.

There are many more attributes than those shown by default, but they are automatically managed by Active Directory. You can’t change them manually. These are attributes built from other attributes.

Among them, one class of attributes is called backlinks. These are attributes that go hand in hand with another attribute, on the same object or on other objects.

For example, the memberOf attribute is a backlink that goes hand in hand with the member attribute. So, as soon as the member attribute of an object is modified (adding or deleting a user or group), this change is automatically reflected in its pair, memberOf, of the object concerned. So if we display these backlinks, we can see the list of groups to which a user belongs.

Well, guess what, the same applies to PSOs. We’re not allowed to list PSOs with a standard user, so we can’t read the msDS-PSOAppliesTo attribute present on each PSO. But we’re in luck, there’s a constructed link for this attribute, and it’s called msDS-ResultantPSO. It’s not a backlink, because it’s a bit smarter. All users in the domain have this attribute, which is potentially updated every time a PSO is applied to users, but also to groups. If a group is added to a PSO, all members of that group will have their msDS-ResultantPSO attribute updated, provided the PSO has priority. So as well as dynamically resolving group members, this attribute will always contain the name of the effective PSO that applies to each user.

And the icing on the cake in all this is that this attribute can be read on all users in the domain, even with a non-privileged account.

Remember, we applied a PSO to the Domain Admins group. Let’s take a look at the constructed attributes of the adm_pixis account, which is part of this group.

We can clearly see our memberOf backlink, which contains Domain Admins, as well as our msDS-ResultantPSO constructed attribute, which contains the PSO we created and applied to the Domain Admins group.

It’s perfect, now we can see whether or not a user’s password policy is affected by a PSO. Mind you, we still don’t have the right to read the contents of the PSO, at least not as a standard user. But from a password spraying point of view, just knowing that a PSO has been applied to a user is extremely valuable. We can simply ignore all accounts for which a PSO is applied, and restrict our tests to users for whom the effective password policy is the default domain policy.

Tools

I followed this white rabbit of password policies by writing a password spraying tool, in an attempt to minimize the risk of locking accounts during my penetration tests. The tool I’ve developed follows this whole process to determine the list of users and the effective password policy for each of them. If the tool is unable to read the contents of the PSO, it will simply ignore the affected accounts during the spraying.

But I wanted to go a step further. We saw that after a certain period of time, the badPwdCount attribute of users was reset. This certain duration, found in the lockoutObservationWindow attribute on the domain, or in the msDS-LockoutObservationWindow attribute on a PSO, can thus vary from one user to another. The tool I’ve developed takes as input a list of passwords we wish to test on users, and will try all the passwords on all the users, taking care to follow locking thresholds and waiting times for the counter to be reset to zero before continuing with the tests.

To ensure that everything runs smoothly, the tool starts by retrieving the time from the domain controller so that it’s perfectly synchronized, and synchronizes regularly with LDAP (a real user might be trying a wrong password during our password spraying).

In short, I’m very happy to share with you the conpass tool, which is extremely useful for me in penetration testing, and I hope it will be as useful for you.

Be careful, it worked during my last pentests, but it’s still a tool written by a human who makes mistakes, that’s going to test passwords. So it’s possible that there are bugs and that it crashes, or that it locks accounts. So I’m very keen to get feedbacks from tests in controlled environments, to make it as robust as possible.

Tokens ERC20 et ERC721

Mon, 16 Oct 2023 03:13:32 +0000

A large proportion of decentralised applications use tokens to work properly. While coins are inherent to each blockchain (Ether for Ethereum, for example, Sol for Solana, etc.), tokens are tokens that are created on an existing blockchain using smart contracts. So, using a smart contract, it is possible to create a token called a “HackndoToken” whose symbol would be “HND”, for example. This token could exist in a limited number, and we could even ensure that each HND token is unique.

These tokens can be transferred from one address to another, they can be created, destroyed, kept in a “safe”, etc. However, if everyone creates their token in their own way, with their own rules, it would quickly become a merry mess. Some tokens might have a transfer function to transfer a token, others might use send(), sendTo(), transferToken(), or even functionToTransferATokenToSomeoneLikeYouuuuu(). In short, things wouldn’t work out. It wouldn’t be possible to exchange one token for another without listing all the functions of all the existing tokens.

That’s why, as with every emerging technology, a standard must be used to facilitate communication between applications, between tokens. Several improvements to Ethereum (Ethereum Improvement Proposal - EIP) have therefore been proposed in order to define different token standards depending on the application’s needs.

Fungible tokens - ERC20

The improvement proposal #20 describes a “classic” token standard. This proposal has been accepted, and the details of this standard are available at this address. As it was issue #20 that was at the origin of this standardisation, this standard is called ERC20 (ERC for Ethereum Request for Comments).

When I say that it’s a “classic” token, I mean that it’s a token with basic properties. It has a name, a symbol, and can be transferred from one address to another. All tokens (from the same contract) are equivalent, just as two bus tickets from the same town are equivalent. These tickets, like these tokens (or like Ethers), are interchangeable. They are therefore called fungible tokens.

In reality, having a name or symbol isn’t even required. It’s only convenient for humans, to help distinguish between tokens other than by their address. It’s a bit like URLs, which are much easier to remember than IP addresses. These two pieces of information, if used, must be accessible via the following methods:

function name() public view returns (string)
function symbol() public view returns (string)

Another optional information can be provided: the number of decimals supported by the token. If the “HND” token has 8 decimals, then in order to have 1 HND you actually need to have 100,000,000! It’s like euros: you need 100 cents to get one full euro. You would then need 10^8 fractions of HND to get 1 HND. So if a user has 150,000,000 HND, a web application will indicate that they have 1.5 (150,000,000 / 100,000,000).

This information can then be accessed using the following method:

function decimals() public view returns (uint8)

In addition to these three pieces of information, which make them easier to use by humans, these tokens must implement the following functions:

// This function must return the total supply of tokens (whether or not they have been distributed).
// If there is only 1 HND, with 8 decimals, this function returns 100,000,000.
function totalSupply() public view returns (uint256)

// Returns the number of tokens owned by an address.
function balanceOf(address _owner) public view returns (uint256 balance)

// Transfers tokens to another address.
function transfer(address _to, uint256 _value) public returns (bool success)

// Transfers tokens from a source address to a destination address.
// For this to work, the source address must have given prior authorisation to the // address performing the transfer.
// performing this `transferFrom` to transfer tokens (it would be too easy otherwise ;))
function transferFrom(address _from, address _to, uint256 _value) public returns (bool success)

// This function can be used to delegate the spending of a set number of tokens to an account.
// of tokens. This function must be called before the delegated account can
// use the `transferFrom()` function.
function approve(address _spender, uint256 _value) public returns (bool success)

// This function returns the number of tokens that an account can spend on behalf of another account.
function allowance(address _owner, address _spender) public view returns (uint256 remaining)

Example

With all these functions in mind, you can create a brand new token from scratch with Solidity!

Please note that this example is given for information only. It is absolutely not suitable for production.

// SPDX-License-Identifier: GPL-3.0

pragma solidity >=0.8.2 <0.9.0;

contract HackndoToken {

    // When state variables are declared as public, getters are automatically created by Solidity
    // For example, we can call the HackndoToken.symbol() function;
    string public symbol;
    string public name;
    uint8 public decimals;
    uint256 public totalSupply;

    // Find out how many tokens each address has
    mapping(address => uint) private balances;

    // Allows you to find out which address has authorised which addresses to spend how many tokens on its behalf
    // For example :
    // allowed[address1][address2] = 10
    // This means that address2 can spend 10 tokens from address1, instead of address1.
    mapping(address => mapping(address => uint)) private allowed;

    // Events must be emitted for certain actions
    event Transfer(address indexed from, address indexed to, uint tokens);
    event Approval(address indexed tokenOwner, address indexed spender, uint tokens);

    // When the contract is created, information about the tokens will be supplied
    constructor(string memory _symbol, string memory _name, uint8 _decimals, uint256 _totalSupply) {
        symbol = _symbol;
        name = _name;
        decimals = _decimals;
        totalSupply = _totalSupply;
        balances[msg.sender] = totalSupply;
        emit Transfer(address(0), msg.sender, _totalSupply);
    }

    // Returns the number of tokens owned by an address.
    function balanceOf(address _address) public view returns (uint balance) {
        return balances[_address];
    }



    // Transfers tokens to another address
    function transfer(address to, uint value) public returns (bool success) {
        // To transfer tokens, you need to have enough of them
        require(balances[msg.sender] >= value, "INSUFFICIANT_FUNDS");
        
        // The tokens are deleted from the sender
        balances[msg.sender] = balances[msg.sender] - value;

        // And we add them to the receiver
        balances[to] = balances[to] + value;

        // An event is sent to register the transfer
        emit Transfer(msg.sender, to, value);
        return true;
    }

    // Delegate the spending of a defined number of tokens to an account
    function approve(address spender, uint value) public returns (bool success) {
        // Whoever calls the function authorises "spender" to spend "value" tokens on their behalf
        allowed[msg.sender][spender] = value;
        emit Approval(msg.sender, spender, value);
        return true;
    }

    // Spend tokens on behalf of another account, provided that the person calling the function has been authorised to do so
    function transferFrom(address from, address to, uint value) public returns (bool success) {
        // We check that the person calling the function is authorised to spend the same number of tokens
        require(allowed[from][msg.sender] >= value, "NO_APPROVAL");

        // Then we check that the account spending the tokens has enough tokens
        require(balances[from] >= value, "INSUFFICIANT_FUNDS");

        // Authorisation is decremented by the number of tokens used
        allowed[from][msg.sender] = allowed[from][msg.sender] - value;

        // Then we exchange the number of tokens from one account to another
        balances[from] = balances[from] - value;
        balances[to] = balances[to] + value;
        emit Transfer(from, to, value);
        return true;
    }

    // Allows you to know how many tokens "spender" can use on behalf of "tokenOwner".
    function allowance(address tokenOwner, address spender) public view returns (uint remaining) {
        return allowed[tokenOwner][spender];
    }
}

This contract can be compiled and deployed on the Ethereum blockchain to create a new token. Incredible, isn’t it?

This type of token (this example, or another) can represent just about anything. It could be the equivalent of money in a video game, skill points, shares in a company (centralised or not), etc.

In this example, we’ve written a token from scratch. However, to avoid mistakes, and to ensure that standardisation goes as smoothly as possible, you shouldn’t reinvent the wheel, and you should use an audited and proven version, such as the one proposed by OpenZeppelin.

Non-Fungible Tokens (NFT) - ERC721

Non-Fungible Tokens (NFT) are a category of tokens that are specifically non-fungible. This means that two tokens, even though they come from the same smart contract, are different. This concept can be compared to cards that can be collected. Although the same company publishes cards representing, for example, the best hackers on the planet, each card represents a particular hacker. It may be from the same collection, but it is not equivalent to another card representing another person.

To standardise tokens with this notion of uniqueness, a new Ethereum enhancement request has been made, the #721, and this new standard is described in the Ethereum documentation, called ERC721.

To ensure that each token from the same smart contract is unique, a new variable is introduced, tokenId. This variable must be unique for each token in a smart contract in order to comply with the ERC721 standard.

In addition to this variable, the following methods must be implemented:

// Allows you to find out how many NFTs an account has
function balanceOf(address _owner) external view returns (uint256);

// Determines the owner of a particular NFT
function ownerOf(uint256 _tokenId) external view returns (address);

// Send an NFT from one address to another, making sure that the destination address
// is either an EOA account or a contract that can handle NFTs
// We'll explain how in the rest of this article
function safeTransferFrom(address _from, address _to, uint256 _tokenId, bytes data) external payable;
function safeTransferFrom(address _from, address _to, uint256 _tokenId) external payable;

// Send an NFT from one address to another
function transferFrom(address _from, address _to, uint256 _tokenId) external payable;

// Authorise an account to manage an NFT on its behalf
function approve(address _approved, uint256 _tokenId) external payable;

// Authorise an account to manage ALL its NFTs on its behalf
function setApprovalForAll(address _operator, bool _approved) external;

// Check who can manage a particular NFT
function getApproved(uint256 _tokenId) external view returns (address);

// Check whether an account has full delegation for another account's NFTs
function isApprovedForAll(address _owner, address _operator) external view returns (bool);

There are a number of functions that are very similar to ERC20 token functions (balanceOf, transferFrom, approve for example). However, two other methods deserve a little more detail.

safeTransferFrom

The first is safeTransferFrom(). This method exists to prevent NFTs being sent to contracts which do not know how to handle NFTs. If this were the case, as the destination contract had not been created to handle NFTs, there would be no function to handle the newly received NFT. This would mean that this NFT could not be bought by anyone, or recovered in any way whatsoever. It would be locked into this contract forever. It’s easy to imagine that, when you offer a limited collection of something, you want to avoid losing it in the wild, unusable and non-exchangeable.

To avoid this kind of problem, when the safeTransferFrom() function is called to send tokens to a contract, the destination contract must use a special function, onERC721Received.

function onERC721Received(
  address operator,
  address from,
  uint256 tokenId,
  bytes calldata data
) external returns (bytes4);

This function will be called by the token contract, and expects a very specific response. If this function does not exist in the destination contract (or if the function exists but does not return what is expected) then the NFT transfer will be cancelled. So, to receive NFTs via safeTransferFrom, a contract must have explicitly included this function. As this function only exists to validate a safeTransferFrom, as a general rule, if a contract contains this function, it means that it is also capable of managing NFTs.

The presence of onERC721Received does not guarantee that the contract can handle NFTs. You could very well create a contract that only implements onERC721Received, and nothing else. The call to this callback is more of a safeguard against silly errors.

setApprovalForAll

The other function that deserves attention is setApprovalForAll, simply because it can be dangerous. When a user uses this function to approve a destination address, it allows the destination to manage ALL of the user’s NFT collection. When we say “manage”, we mean that the destination can send the user’s NFTs to arbitrary destination addresses. He could send them to the address null (0x0), which would cause these NFTs to be lost forever, or even send them to himself. Once the transfer is complete, the user has no way of recovering them.

This function is dangerous and should only be used if you have absolute trust in the recipient (if it’s an EOA) or absolute understanding of the code (if the destination is a smart contract).

Example

Here is an example of an implementation of ERC721 from scratch showing a simplistic implementation of the functions.

Please note that this example is for illustrative purposes only. It is absolutely not suitable for production.

// SPDX-License-Identifier: GPL-3.0

pragma solidity >=0.8.2 <0.9.0;

// This interface must be declared in order to be able to call the callback when
// a safeTransferFrom
interface IERC721Receiver {
    function onERC721Received(
        address operator,
        address from,
        uint256 tokenId,
        bytes calldata data
    ) external returns (bytes4);
}

contract HackndoToken {

    // When state variables are declared and made public, getters are automatically created by Solidity
    string public symbol;
    string public name;

    uint private tokenId;

    // Each NFT is assigned an owner
    mapping(uint256 => address) private owners;

    // Each address is assigned the number of NFTs it owns
    mapping(address => uint256) private balances;

    // Allows you to find out which address has delegation rights over which NFT
    mapping(uint256 => address) private tokenApprovals;

    // Addresses which have full rights over the NFTs of other addresses
    mapping(address => mapping(address => bool)) private operatorApprovals;



    // When the contract is created, information about the NFT will be supplied
    constructor(string memory _symbol, string memory _name) {
        symbol = _symbol;
        name = _name;
    }

    // This function is not included in the standard, but it allows you to create
    // NFT at will, with no restrictions! Enjoy :)
    function mint(address _to) public {
        owners[tokenId] = _to;
        balances[_to]++;
        // As an NFT is created, the identifier must be incremented
        // so that the next NFT is different
        tokenId++;
    }

    // Allows you to find out how many NFTs an account has
    function balanceOf(address _address) public view returns (uint balance) {
        return balances[_address];
    }

    // Determines the owner of a particular NFT
    function ownerOf(uint256 _tokenId) external view returns (address) {
        // You cannot have a tokenId greater than the last one created
        require(_tokenId < tokenId, "NOT_EXISTANT");
        return owners[_tokenId];
    }

    // Send an NFT from one address to another, while ensuring that the destination address
    // is either an EOA account or a contract that can handle NFTs
    function safeTransferFrom(address _from, address _to, uint256 _tokenId, bytes memory data) public payable {
        transferFrom(_from, _to, _tokenId);

        // If it's a contract, call the onERC721Received() callback.
        // If there is an error, or if the return value is not what was expected, we revert();
        // -> If there is a revert(), then the transferFrom() function previously called will be cancelled
        if (_to.code.length > 0) {
            try IERC721Receiver(_to).onERC721Received(msg.sender, _from, _tokenId, data) returns (bytes4 retval) {
                if (retval != IERC721Receiver.onERC721Received.selector) {
                    revert();
                }
            } catch (bytes memory) {
                revert();
            }
        }

    }

    function safeTransferFrom(address _from, address _to, uint256 _tokenId) external payable {
        safeTransferFrom(_from, _to, _tokenId, "");
    }

    // Send an NFT from one address to another
    function transferFrom(address _from, address _to, uint256 _tokenId) public payable {
        // Either it owns the NFT, or it has been approved for this NFT, or it has been approved for all the NFTs (including this one) in another account
        require(owners[_tokenId] == msg.sender || tokenApprovals[_tokenId] == msg.sender || operatorApprovals[_from][msg.sender], "NO_APPROVAL");
        // The transfer source must own the NFT
        require(owners[_tokenId] == _from, "NOT_OWNER");
        owners[_tokenId] = _to;
    }

    // Authorise an account to manage an NFT on its behalf
    function approve(address _approved, uint256 _tokenId) external payable {
        require(owners[_tokenId] == msg.sender, "NOT_OWNER");
        tokenApprovals[_tokenId] = _approved;
    }

    // Allow an account to manage ALL its NFTs on its behalf
    function setApprovalForAll(address _operator, bool _approved) external {
        operatorApprovals[msg.sender][_operator] = _approved;
    }

    // Check who can manage this particular NFT
    function getApproved(uint256 _tokenId) external view returns (address) {
        return tokenApprovals[_tokenId];
    }

    // Check whether an account has full delegation for another account's NFTs
    function isApprovedForAll(address _owner, address _operator) external view returns (bool) {
        return operatorApprovals[_owner][_operator];
    }
}

As with ERC20, there is a version of ERC721 proposed by OpenZeppelin which means you don’t have to start from scratch and can use a solid, tried and tested code base.

Conclusion

These two standards are the most widely known and used, but they are far from being the only ones in existence. In fact, tokens can be used for so many applications that standards develop (and sometimes die) as new ideas for their use are put forward.

Understanding how these tokens work is essential for any good auditor, as they are extremely common in decentralised applications, or dApps.

Sensitive Data in smart contracts

Tue, 03 Oct 2023 08:09:08 +0000

Do you remember the different storage spaces to which the EVM has access? The one comparable to a computer hard disk is the account storage. This is the memory area in which the state of the contract is recorded. But you’ll also remember that the Ethereum blockchain is a decentralised state machine that can be read by anyone. Do you see where I’m going with this? All the data recorded by a smart contract can be read by anyone. If any sensitive data is recorded by a smart contract, we will be able to read it.

Memory reminder

EVM memory is divided as follows:

In the article on EVM, we described the usefulness of the different memory zones and how they are arranged.

What we’re interested in in this article is account storage, the permanent storage of the smart contract’s account. It is in this storage area that the contract will record its variables, which must be persistent on the blockchain. For example, if a smart contract manages registration for an event, it is necessary for the list of registrants to be recorded, and to be able to be modified. It is typically for this type of information that acount storage is used.

As a reminder, here’s what this memory area looks like:

It is organised into slots, which function like an index. There are 2**256 locations, and each location can store 256 bits.

If a contract (written with Solidity) wishes to store variables in this space (which we will call state variables), it must declare them outside the functions.

contract Hackndo {
  /**
   * State variables registered in Account Storage
   */
  uint256 id = 7; 
  uint256 totalAmount = 1000;


  /**
   * Contract code
   */
  constructor() {
    // Code
  }


  function test() external {
    // Local variable (not stored on the blockchain)
    uint256 localVariable = 0;
  }


  function update() external {
    id++;
    totalAmount = 0;
  }
}

The id and totalAmount variables will be stored in this contract’s account storage, and will be accessible by all functions in this contract. If they are updated by a function (such as update()), the contract’s account storage will be updated and these new values will be available for future transactions.

Variable visibility

With Solidity, the visibility of a variable can be defined in three different ways:

public: The variable is readable by other smart contracts. A getter is automatically created. It can therefore be read by calling the id() or totalAmount() function, for example.
internal: The variable can only be read or modified by the contract in which it is defined, or contracts which inherit from this contract. This is the default visibility for variables.
private’: The variable cannot be read or modified by any smart contract other than the one in which it is defined.

The definitions of internal and private variables in the Solidity documentation can be confusing:

Internal state variables can only be accessed from within the contract they are defined in and in derived contracts. They cannot be accessed externally. This is the default visibility level for state variables. Private state variables are like internal ones but they are not visible in derived contracts.

If we were not careful, we might think that by defining an internal or private variable, this variable could not be read by anyone other than the contract itself, or contracts inherited from it, and that we could therefore be storing confidential information.

The internal and private variables are only private within the smart contract. However, their values can be freely read outside the blockchain by anyone, so they do not hide data in this way.

Account storage layout

As an attacker, it is then necessary to understand how variables are stored in account storage.

Storage order

The first thing to understand is that storage variables are stored by the Solidity compiler in the order in which they are declared. In the example given above, the id variable will be stored first, then the totalAmount variable.

If no value is assigned to the variable, it will take the default value 0x00, and its slot is still reserved.

When the smart contract is compiled, the compiler will try to optimise the storage space required. To do this, if variables can fit into the same 32-byte slot, they will be put into the same slot.

For example, if the state variables are as follows:

contract Hackndo {
  /**
   * State variables stored in Account Storage
   */
  uint32 var1 = 7; 
  uint32 var2 = 15;
  uint128 var3 = 10;
  uint128 var4 = 9;
  uint32 var5 = 2;
  uint8 var6 = 3;
}

The size of a slot is 256 bits. The first 3 variables occupy 32+32+128 = 192 bits. The 4th variable cannot be added to the same slot, as there are only 64 bits available. It therefore goes into the second slot, along with the 5th and 6th variables. The size of var4, var5 and var6 is 128+32+8 = 168 bits, which fits into one slot.

This gives the following data in the storage:

# Slot 0
0x00000000000000000000 0000000000000000000000000000000a 0000000f 00000007
# empty var3 var2 var1


# Slot 1
0x00000000000000000000 0003 00000002 00000000000000000000000000000009
# empty var6 var5 var4

Constant & Immutable

With Solidity, the constant and immutable keywords can be used on state variables.

If a variable is defined as constant, a value must be assigned to it when it is declared, and this value can never be changed.
If a variable is defined as immutable, it must be assigned a value, either at declaration time or in the constructor.

What these two types of variable have in common is that all uses of these variables in the code will be replaced by their value by the compiler before the bytecode is saved on the blockchain. So in fact, these notions of constant and immutable don’t exist for EVM. It’s just something practical for developers.

If, for example, we have the following contract:

contract Hackndo {
  uint256 constant MAX_SUPPLY = 1000;
  uint256 immutable DEST_ADDR;

  constructor(address _dest_addr) {
    DEST_ADDR = _dest_addr;
  }

  function someFunc(uint _value) {
    require(_value < MAX_SUPPLY, "MAX_SUPPLY reached");
    require(msg.sender == DEST_ADDR, "Not allowed");

    // Some code
  }
}

Two variables MAX_SUPPLY and DEST_ADDR are declared. However, they will be replaced by their values when the contract is deployed on the blockchain. So finally, if this code is deployed by address 0x1234..., it is exactly equivalent to :

contract Hackndo {

  function someFunc(uint _value) {
    require(_value < 1000, "MAX_SUPPLY reached");
    require(msg.sender == 0x1234..., "Not allowed");

    // Some code
  }
}

From a bytecode point of view, constant and immutable variables don’t exist. So if you see this type of variable in a contract, you mustn’t take them into account when calculating slots.

Storing variables

Now that we’ve clarified which variables are stored in storage, and the optimisation used to limit the size of storage used, let’s look at how the different types of variable are technically stored.

Integers and Booleans

As we saw in the previous examples, integers (and booleans) are simply stored in the corresponding slot. The maximum size of an integer was 256 bits, so it can never be larger than the size of a slot, which is also 256 bits.

Table

When a array has a defined size, then its elements are stored one after the other following the rules already seen.

But a array can have a dynamic size. We are not going to change the slots of all the variables that follow the array every time the size of the array changes. Each element of the array has its own slot in which it is stored.

In this way, only the size of the array is stored in the slot that follows the rules we have described (so if a dynamic array is stored in slot 3, its size will be found in this slot).

To find the first element of the array, calculate keccak256(abi.encode(arrayIndex)) (arrayIndex would be 3 in the previous case). This result is a 256-bit hash, which corresponds to the number of the slot in which the first element of the array is located. The following elements are simply in the following slots.

Mapping

For a mapping, a slot is reserved to determine its base index, but nothing is stored there, unlike arrays where the size is stored.

To access an element in a mapping, you don’t use an index, but the element’s key to find out its value.

To determine where a mapping value is based on its key, you need to calculate the hash that concatenates the key of the element you are looking for and the slot reserved for the mapping (key + slot). The keccak256(abi.encode(key, slot)) function must therefore be applied. As with arrays, this function returns a hash, which corresponds to the slot in which the value of key is located.

String

strings of less than 32 bytes are stored in a slot. The most significant bits are used to store the string, and the least significant bits to indicate the length of the string multiplied by 2 length*2.

If it is 32 bytes or longer, then the slot reserved for the string contains the length of the string multiplied by two, plus 1, length*2+1, and the location of the string is simply the hash of the reserved slot.

For example, if a long string is supposed to be in slot 2, then the address where the string is actually located can be found with the function keccak256(abi.encode(2)).

➜ bytes32 slot = keccak256(abi.encode(2));
➜ slot
Type: bytes32
└ Data: 0x405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace

This technique of storing double the length of the string, or double plus 1, lets you know whether you are storing a string of less than 32 bytes or more than 32 bytes. If the least significant bit of the size is 1, then the string is longer than 32 bytes. Otherwise, it is less than 32 bytes. Removing this bit and dividing the size by 2 gives the actual size of the string.

Structure

Finally, the variables in a structure are stored one after the other, as if they were independent variables. If, in the structure, there are dynamic types (array, mapping etc.), then the rules we have seen apply.

Example

Here’s an example to summarise what we’ve seen so far:

// Definition of a structure
struct Coin {
    string name;
    uint256 price;
}


// Definition of the example contract
contract StorageContract {
    uint256 constant MAX_SUPPLY = 1000;
    address immutable DEST_ADDR;
    uint256 totalSupply = 10;
    string author = "pixis";
    string description = "This is an example of storage layout made by pixis. All details in https://hackndo.com";
    uint[] coinsId = [1,2,10,12];
    mapping (string=>address) accounts;
    Coin coin = Coin("PixCoin", 0x1000);

    constructor() {
        DEST_ADDR = msg.sender;
        accounts["pixis"] = msg.sender;
        accounts["empty"] = address(0x0);
    }
}

When this contract is deployed, this is what the storage looks like :

Let’s try to break this down. Firstly, the first two variables MAX_SUPPLY and DEST_ADDR are not stored in storage, so no slots are reserved for them.

Next, the following variables are assigned a slot, in the order in which they are declared.

To perform the calculations, I use chisel from the Foundry suite.

totalSupply is a 256-bit integer, so an entire slot is reserved for it, slot 0. Its value is 10, so 0x0a
author is a string of less than 32 bytes. It is therefore stored in the next slot, slot 1, at the level of the most significant bits. Its size, multiplied by two ( 5*2 = 10 = 0x0a) is stored in the least significant bits.
description is a string of 86 bytes, so greater than 32 bytes. Thus, its slot 2 contains double its size, to which 1 is added (remember, by adding 1, it indicates that the string is longer than 32 bytes), so 86*2+1 = 173 = 0xad. The slot containing the string corresponds to the hash of the string’s slot, so 2. However keccak256(abi.encode(2)) = 0x405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace so the slot 0x405787fa12a823e0f2b7631cc41b3ba8828b3321ca811111fa75cd3aa3bb5ace contains the string.
coinsId is an array containing 4 elements. Its size 0x04 is therefore specified in its slot 3. The slots of these 4 elements are calculated as follows:
- Index 0: keccak256(abi.encode(3)) = 0xc2575a0e9e593c00f959f8c92f12db2869c3395a3b0502d05e2516446f71f85b. For the other elements, the slot is incremented by 1 each time.
- Index 1: 0xc2575a0e9e593c00f959f8c92f12db2869c3395a3b0502d05e2516446f71f85c
- Index 2: 0xc2575a0e9e593c00f959f8c92f12db2869c3395a3b0502d05e2516446f71f85d
- Index 3: 0xc2575a0e9e593c00f959f8c92f12db2869c3395a3b0502d05e2516446f71f85e
accounts is a mapping whose slot is 4. It’s noticed that this slot is empty, that’s normal. The size of the mapping is not stored. To find the value of a particular key, the function keccak256(abi.encodePacked(key, slot)) should be used so:
- accounts["pixis"] is found at the slot keccak256(abi.encodePacked("pixis", uint(4))) = 0x47e3196153c18a6193d6b7b92ecf7ea03bc91cce35ccd718094e10f1c50bd1e9
- accounts["empty"] is found at the slot keccak256(abi.encodePacked("empty", uint(4))) = 0xace73dd693559189ef5ccbbc8f81155ea53ec7259b948d81d0791cf64125f053
coin is a structure containing two elements. They are therefore positioned in the slots 5 (name, less than 32 bytes) and 6 (price, worth 0x1000).

With all these explanations, we are able to understand the entire account storage of this contract, once deployed.

Memory reading

It’s great, we are able to read and understand the storage space of contracts, but concretely, how do we access the storage space of a contract already deployed on the blockchain?

Different tools allow reading the slots of a contract’s storage. Personally, I use the cast tool from the foundry suite.

Indeed, when you install foundry on your machine, different tools are installed:

Forge: Framework for performing tests on Ethereum
Cast: Tool for interacting with smart contracts and the blockchain
Anvil: Local Ethereum node
Chisel: REPL tool for quickly executing Solidity code

The cast tool is very handy for reading a contract’s slots. The syntax is as follows:

cast storage 0xcontract_address slot_number [--rpc-url RPC_URL]

For example, to read the slot 0 of the contract at address 0x099A3B242dceC87e729cEfc6157632d7D5F1c4ef on Ethereum (random contract), the following command line can be used:

cast storage 0x099A3B242dceC87e729cEfc6157632d7D5F1c4ef 0 --rpc-url https://eth.llamarpc.com 
0x0000000000000000000000000000000000000000000000000000000000000001

So, the value 0x01 is in slot 0 of the contract. We can loop to read the first 6 slots:

for I in {0..5} 
do
    echo "SLOT $I: " $(cast storage $CONTRACT_ADDR $I --rpc-url $RPC_URL)
done
SLOT 0:  0x0000000000000000000000000000000000000000000000000000000000000001
SLOT 1:  0x0000000000000000000000000000000000000000000000000000000000000000
SLOT 2:  0x00000000000000000000000000000000000000000000000000c6645100000000
SLOT 3:  0x0000000000000000000000000000000000000000000000000000000000000205
SLOT 4:  0x000000000000000000000000000000000000000003f806d77433774f8c683600
SLOT 5:  0x0000000000000000000000000000000000000000000000000000000000c6647c

Putting it into practice

A contract is deployed at address 0x84229eeFb7DB3f1f2B961c61E7CbEfd9D4c665E3 on the Sepolia test network.

This contract is a game whose code is:

pragma solidity ^0.8.9;

contract GuessingGame {
    address public owner;
    mapping(address => bool) public hasGuessed;
    uint256 private secretNumber; // Declared as private. Is it really private?
    
    
    constructor() {
        owner = msg.sender;
        secretNumber = 12345; // This is not the real number
    }

    function guess(uint256 _number) public {
        if (_number == secretNumber) {
            hasGuessed[msg.sender] = true;
        }
    }

    function isWinner(address _addr) public view returns (bool) {
      return hasGuessed[_addr];
    }
}

The goal is to call the function guess() by providing a number. If you hit the right number, you win, and you can prove it with the isWinner() function.

As we have seen in this article, the variable secretNumber has been declared as private, but that will not prevent us from retrieving this value. For this, let’s use the cast tool.

To encourage you to try, the result provided below is not the real result. It’s up to you to find the real secret value! The logic remains the same.

RPC_URL=https://rpc2.sepolia.org                                        
CONTRACT_ADDR=0x84229eeFb7DB3f1f2B961c61E7CbEfd9D4c665E3

for I in {0..3}
do
    echo "SLOT $I: " $(cast storage $CONTRACT_ADDR $I --rpc-url $RPC_URL)
done

# Output
SLOT 0:  0x00000000000000000000000031d6273610256e6cefd6f26a503c72bb2bdcfe15
SLOT 1:  0x0000000000000000000000000000000000000000000000000000000000000000
SLOT 2:  0x0000000000000000000000000000000000000000000000000000000042424242
SLOT 3:  0x0000000000000000000000000000000000000000000000000000000000000000

We see that the first three slots are used. The first corresponds to the first state variable, that is the owner address. The second variable seems empty, but that’s normal. This is the slot used by the hasGuessed mapping. secretNumber is recorded in the 3rd slot, and its value is 0x42424242.

Congratulations, you have discovered a secret variable in a contract deployed on an Ethereum network!

To interact with the contract, still with the cast utility, here’s how to proceed:

# To create a transaction, we use cast send
# To be able to sign the transaction, the private key must be provided.
cast send $CONTRACT_ADDR "guess(uint256)" "10" --private-key 0xabcdabcd...abcd --rpc-url $RPC_URL

# To read information without modifying the storage, we use cast call.
# isWinner() writes nothing in the storage, so no need to give it a private key. It's just information reading.
# If the output is 0, your address has still not found the right number.
# If the output is 1, congratulations, you have found the secret number!
cast call $CONTRACT_ADDR "isWinner(address)" "your address" --rpc-url $RPC_URL

It’s your turn to play!

Ethereum Virtual Machine

Wed, 19 Jul 2023 08:12:43 +0000

Ethereum Virtual Machine (EVM) is a virtual machine used to manage transactions on the Ethereum blockchain via smarts contracts. It’s an essential component of Ethereum, which we’re going to try and understand together.

EVM

To execute smart contracts (programs in Ethereum’s world), rules must be followed. These rules are partly described in Ethereum’s Yellow Paper, and can be implemented by anyone in any language. There is a python version of EVM (py-evm), a Rust version (revm), and a Go version (go-evm). This list is by no means exhaustive.

Opcodes

One of EVM’s key features (like any computer) is the ability to read and execute instructions, or opcodes. Ethereum instructions are described on the official Ethereum website, Opcodes for the EVM. The website evm.codes is also very useful.

This is the kind of code that is understood by the EVM. It is generated when a high-level language is compiled. One of the most widely used languages for writing smart contracts is Solidity.

Here’s a very simple example of a smart contract written in Solidity.

// SPDX-License-Identifier: GPL-3.0

pragma solidity 0.8.18;

contract HackndoMembers {
    // Declaring persistent variables in the blockchain
    address public owner;

    address[] public members;
    uint private memberCount;

    // Constructor, executed when smart contract is deployed
    constructor() {
        owner = msg.sender;
    }

    // Public function to register as a member
    function becomeMember() external {
        members.push(msg.sender);
        memberCount++;
    }

    // Public function to find a member
    function getMember(uint _id) external view returns(address member) {
        require(_id < memberCount, "id too big");
        require(members[_id] != 0x00, "Not a member");

        member = members[_id];
    }

    // Function only available to the smart contract creator to delete a member
    function removeMember(uint _id) external {
        require(msg.sender == owner, "Owner only");
        members[_id] = address(0x0);
    }
}

Once compiled, this program will be a sequence of instructions, or opcodes, understood by the EVM. The solc tool can be used to compile Solidity.

$ solc contract.sol --bin        

======= contract.sol:HackndoMembers =======
Binary:
608060405234801561001057600080fd5b5033600080610100[...]

It also shows the generated opcodes.

$ solc contract.sol --opcodes

======= contract.sol:HackndoMembers =======
Opcodes:
PUSH1 0x80 PUSH1 0x40 MSTORE CALLVALUE DUP1 ISZERO PUSH2 [...]

Some of these instructions are used to execute mathematical operations, such as add, sub, mul or div. Others are used to compare elements, such as lt (Lower Than), gt (Greater Than) or eq.

It is possible to read from and write to different storage areas, such as memory with mLoad, mStore, or storage with sLoad, sStore for example.

The stack (another memory area) is managed with opcodes such as push1, push2, …, push32, and pop.

These different types of storage will be discussed later in this article.

A contract can make calls to other functions, potentially other contracts, via call, staticcall and delegatecall.

Finally, the revert instruction can be used to make a kind of exception that terminates the current call. In most cases, the transaction will be considered invalid, and no changes will be made.

These examples are by no means exhaustive, but they give an idea of what the EVM has to deal with when a smart contract is executed.

Gas

Each instruction executed on the nodes has a price, the unit of which is the gas. For example, executing an add costs 3 gas, while a pop costs just 2.

When calling a smart contract function, a user must pay the price required to execute the instructions. He must therefore provide sufficient gas during his transaction. If he has supplied too much, it’s no problem - the remaining gas will be refunded.

If, on the other hand, he has not supplied enough, the instructions will be executed until the gas is exhausted. When this happens, the transaction is cancelled, and the gas supplied by the user is lost. Although the transaction is cancelled, it still took resources to detect it, so it’s too late.

This notion of gas was introduced to prevent resources from being used unnecessarily, in particular to avoid infinite loops or attacks that would clog up the network. In fact, there is a maximum number of gas possible in a single block (currently 30 million gas).

Solidity

For the rest of this article, bear in mind that EVM, in the end, just executes opcodes, one after the other. It also offers various empty storage spaces that can be used, and that’s it. How these opcodes are organized, or how the data is structured, is up to the compiler.

What we’re going to look at in this article concerns the Solidity compiler (and language). Other language compilers often use Solidity as a reference and follow the same conventions, but this is not always the case.

Global variables

When a smart contract is written with Solidity, there are three global variables accessible to the smart contract, providing information about the context in which it is executed:

Block (block): This variable contains information about the block in which the transaction was validated. This includes the block number, the time it was added to the blockchain, and its hash.
Transaction (tx): Information relating to the current transaction is available in this variable. This is where we’ll know, for example, who initiated the transaction (who may be different from who initiated the last message), so it will always be an EOA.
Message (msg): Several messages can be sent within a transaction. In these messages, you can find out who sent the message, how many Ethers were supplied, the data included in the message, and so on. Depending on the context and the message, the msg variable may change. For example, when a contract calls another contract, the msg.sender attribute will be modified.

Storage

The smart contract code (made up of opcodes such as the ones we’ve introduced) needs to be stored somewhere, as do the contract variables and other temporary or permanent data required for proper execution. For this purpose, the EVM has various types of storage, permanent or not, for different purposes.

Permanent storage

There are two types of persistent storage. These are the places where information is stored by nodes, and persistent during transaction execution. So, when a transaction is completed, this storage will be saved, and can be used for the next transaction. How convenient!

Bytecode

The smart contract code is permanently stored, but cannot be modified. It’s read-only. If an issue is detected in the smart contract after it has been deployed, it’s too late. You have to deploy a new smart contract with its fix, and warn users that the smart contract address has changed.

There are ways of dealing with this problem with proxy smart contracts, but that’s not the topic, and these contracts can also have bugs.

Account storage

The persistent storage location for smart contracts is account storage. It’s a bit like a computer hard drive. We talked about this in Ethereum article. In the world state (Ethereum’s global state), each address is associated with various elements, such as the account’s Ether balance, but also, in the case of smart contracts, a “storage space” specific to the smart contract.

In practical terms, this account storage is a key/value database. The key is a 256-bit value, and so is the value. We can therefore store 2**256 keys, which should be more than enough. For the sake of clarity, we can also think of this storage as an array of 2**256 rows, and each row can be assigned a 2**256 bits value.

Before anything is executed, this array is empty - it’s all zeros. So each contract has, by default, an array of 2**256 lines, and each line has 2**256 bits set to zero.

Generally speaking, the first slots (or rows) of a Solidity contract contain the contract’s state variables.

Let’s take the following example:

contract Hackndo {
    /**
     * State variables
     */
    uint256 id = 7; 
    uint256 totalAmount = 1000;

    /**
     * Contract code
     */

    constructor() {
        // Code
    }

    function myFunction() external {
        // Code
    }
}

Once the contract has been created, the account storage will contain the following key/values:

When referring to a key, the term slot is often used. In the following example, slot 0 is the id variable and slot 1 is associated with the totalAmount variable.

Optimisation

The declared variables were uint256, i.e. 256 bits, which took up an entire slot, but if smaller variables are used, the storage will be optimized by the Solidity compiler. If two variables fit into one slot, then they will be put into that same slot. We’ll see this in details in another article.

Other types

In this account storage area, you can store not only integers, but also strings, arrays, mappings and so on. Each type of variable has its own storage rules, which are managed by the Solidity compiler so that they can be retrieved. Here’s a quick overview:

When a array is stored, the size of the array is stored at an index which follows the previous rule. To find the array Nth element, you need to compute keccak256(abi.encode(arrayIndex))+N.

keccak256 is a hash function (old SHA3 version). abi.encode is used to encode information in order to transform potentially complex data structures (such as arrays) into a sequence of bytes, enabling a hash function to work correctly.

For a mapping (a key-value association), a slot is reserved to determine its base index (but nothing is stored there, unlike arrays for which the size is stored), then to determine where a value in the mapping is located, the function keccak256(abi.encode(key, mappingIndex)) must be applied. It returns the index where the key value is located.

strings of less than 32 bytes are stored in a slot. The most significant bits are used to store the string, and the least significant bits to indicate its length. If it’s 32 bytes or longer, then the same mechanism as for arrays applies.

Finally, variables in a structure are stored one after the other, as if they were independent variables. If, in the structure, there are dynamic types (array, mapping etc.), then the rules we’ve seen apply.

Volatile storage

Volatile memory is memory which, once the contract has been executed, is erased, leaving no trace of it. You could compare this memory to RAM (random access memory) in a computer.

Stack

The stack is a LIFO (Last In, First Out) memory location.

This means that the last element pushed on the stack will be the first element to be unstacked. To better understand this, imagine a stack of plates. If you stack plates on top of each other, you’ll have to remove the last plate placed on top, then the second-to-last plate, etc., before you can retrieve the first plate placed on top.

This memory area is used by the compiler to store temporary information, such as a function’s local variables, or opcodes arguments, for example. Typically, all smart contracts compiled with Solidity start with these 3 instructions to store the value 0x80 at memory address 0x40.

PUSH1 0x80 // destination
PUSH1 0x40 // value
MSTORE // mstore(destination, value)

mstore function arguments are pushed onto the stack in the reverse order of their use. In fact, the first element to be popped will be the last element pushed. So, first the value 0x80 is pushed, then the destination 0x40. When mstore is executed, 0x40 (the destination) will be popped, followed by 0x80 (the value).

This is a memory area that changes a lot as a program is executed. Up to 1024 elements of 256 bits (32 bytes) can be stored here.

Please note that only the first 16 elements of the stack can be used to perform operations, call functions, etc. This means, for example, that a function cannot have more than 16 arguments, or more than 16 local variables.

Memory

The memory of a smart contract is a large memory area accessible for reading and writing in no predefined order, unlike the stack. It can store any size of information, from one byte up to 32 bytes at a time. On the other hand, information can only be read by 256 bits (32 bytes). This is where you’ll find variables with dynamic sizes, such as arrays or mappings, but you can also store integers or Booleans.

Addressing is done on 32 bytes. So, theoretically, you can store up to 2**256 bits of information. In practice, this is mainly to avoid collisions when storing dyamically-sized data. We use the hash of certain elements to decide on the storage destination. There’s plenty of time to win the lottery before two hashes in a 2**256 space are close!

Reserved space

The first two bytes (at addresses 0x00 and 0x20) are used by the compiler to perform temporary calculations or operations.

The third location (0x40) contains a pointer to the next free, usable memory area. This is the free memory pointer.

In fact, it is this pointer that is initialized at the start of every contract compiled with Solidity as we saw earlier in this article. The following operations store 0x80 at address 0x40.

PUSH1 0x80
PUSH1 0x40
MSTORE

So the next area free for memory allocation is located at address 0x80. And why not address 0x60? Because this address is also special: it’s always 0. It can be copied to initialize an array, for example.

Data storage

Simple formats such as integers are simply stored at the address assigned to them.

For strings, when an address is assigned to store them, the length of the string is stored in the 256 bits starting at that address, then the string is stored.

For arrays, a placeholder corresponding to the number of elements is reserved, and the array elements are added one after the other.

A structure is organized in the same way as an array.

Calldata

When calling a smart contract function, the call must be created by the client before sending the transaction, i.e. before the EVM is instantiated anywhere. The function’s parameters cannot therefore be in a stack or in the EVM’s memory.

The function, and its arguments, are sent in the data field of the transaction, as we briefly saw in the article on Ethereum. When the contract is actually instantiated and executed in the Ethereum virtual machine, what has been sent in data will be copied into the memory area called calldata.

This memory area, calldata, is used when an Ethereum client calls a function, but not only. It is used every time a message is sent, whether from an EOA to a contract, or from a contract to a contract.

From a memory point of view, calldata is very similar to memory.

It is linear
Addressing is byte-by-byte.
Only 32 bytes can be read per call.

However, unlike memory, this memory area is read only. It cannot be written to. The EVM is responsible for copying the parameters sent by the message source.

Function selector

The first 4 bytes are reserved for the function selector. As explained in the article on Ethereum, the function selector is calculated by hashing the function signature, and keeping only the first 4 bytes.

For example, let’s take the following function:

function getItemValue(string calldata _itemName, uint256 _itemId) public returns(uint256 value) {
  // Function code
}

Function signature is:

getItemValue(string,uint256)

And its selector:

bytes4(keccak256("getItemValue(string,uint256)"));
// Output:
0xc2e58fec

The rest of this memory area is dedicated to function arguments.

Function arguments storage

Simple formats such as integers are stored as is.

For strings, we store the offset of where the string actually is. This offset is used to find the string, starting with its size (on 256 bits) and then the string itself.

For arrays, similarly, we store the offset where the array is located. This offset is then used to store the array elements.

A structure is organized in the same way as an array.

Let’s take the same example as in the previous article about Ethereum:

getItemValue("pixis", 8);

calldata value will be:

0xc2e58fec0000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000057069786973000000000000000000000000000000000000000000000000000000

This can be broken down as follows:

PC - Program Counter

For the record, there’s also a memory area called the Program Counter or PC. For those familiar with the Intel world, this is the equivalent of “EIP” (or “RIP”) register. It’s a memory area containing the address of the next instruction to be executed. It allows the virtual machine to know where to execute the next opcode. Often, this address increases little by little, and sometimes, when there’s a jump, the destination of the jump is assigned to the PC, so that the next instruction executed will be the jump destination.

Gas

Finally, the EVM keeps track of the number of consumed gas, to check that there is sufficient gas supplied by the user.

Calls

Having reviewed the different memory zones that enable the EVM to run, we’ll finish by talking about the different types of calls that can be made to a smart contract to execute code. These calls are used to execute a smart contract function, with arguments if necessary.

Each type of call has its own specificities. To understand what we’re talking about, we first need to explain that a contract is executed in a certain context. Sometimes, when a function is called, a new instance of EVM is deployed to execute the function code. Sometimes the memory areas are different, sometimes shared. Global information (such as the message source) may or may not also vary, depending on the type of call.

We’ll summarize the details of each call in a table below.

Internal calls

The simplest are internal calls. This is what happens when a smart contract calls one of its own functions, or a function from a contract it inherits. In opcode terms, when an internal call is made, a jump is executed. There is no change of context, we remain in the same contract, in the same virtual machine instance. The called function shares the same information and storage areas as the calling function.

Here are two examples of internal calls, one for a function from the same contract (functionA()) and the other calling a function from a parent contract (functionParent()).

contract Parent {
    function functionParent() internal pure {
    }
}

contract Child is Parent {
    function functionA() internal pure {

    }

    function functionB() external pure {
        // Internal call to a function within the same contract
        functionA();
    }

    function functionChild() external pure {
        // Internal call to a function in the inherited contract
        functionParent();
    }

The contents of functionA() could have been put into functionB(), it wouldn’t have made much difference.

External calls

External calls are more interesting. They allow you to call functions from other contracts. There are 3 different types of external call.

In fact, there’s a 4th, callcode, but it’s been deprecated in favor of delegatecall, so we won’t talk about it here.

call

The call opcode is the basic call. It allows you to call a function from another contract. This function will be executed in a new EVM instance, with its own memory zones (stack, memory, etc.). The called code can then do as it wishes, modify its own memory, update its variables, etc. Understand, however, that the variables of the called contract are completely independent of the variables of the calling contract. Good fences make good neighbors.

In addition, message data is updated. Thus, the originating address (msg.sender) becomes that of the calling contract, and the value included in the message (msg.value) is also updated.

You can also send Ethers via a call.

Here’s an example:

contract ContractA {
    uint public callCounter;
    function functionA() external payable {
        callCounter++;
    }
}

contract ContractB {

    ContractA contractA = new ContractA();
    
    function functionB() external {
        // call because functionA modifies information in storage, in this case its "callCounter" variable
        contractA.functionA();
    }
}

It is possible to use the call function explicitly, as follows:

(bool success,bytes memory data) = address(contractA).call{value: 0.1 ether}(abi.encodeWithSignature("functionA()"));

The call will return a boolean status on the successful execution of the call as well as the data optionally returned by the called function. Note also that, in this example, we have sent 0.1 ether to the called contract.

staticcall

staticcall is in every respect similar to call, but the function called cannot make any modifications to the blockchain, neither its storage nor its ether balance. It’s a kind of read-only call.

contract ContractA {
    function functionA() external view {
        // Some code
    }
}

contract ContractB {

    ContractA contractA = new ContractA();
    
    function functionB() external view {
        // staticcall because functionA is declared as "view", so it won't modify its storage
        contractA.functionA();
    }
}

As this call cannot modify the blockchain, the balance of the called contract cannot be modified. It is therefore not possible to send Ethers via this call. It is also possible to call the staticcall function explicitly, as follows:

(bool success,bytes memory data) = address(contractA).staticcall(abi.encodeWithSignature("functionA()"));

delegatecall

The delegatecall call is very special. It can be extremely useful, but also extremely dangerous. Whereas with the call and staticcall calls, the memory areas were clearly separated between the caller and the called party, this is not completely the case with the delegatecall call.

In this case, all volatile memory areas (stack, memory, PC) are specific to the called contract, contract B, however:

The reads and writes to storage will be done in the storage of contract A.
The message origin address (msg.sender) and value (msg.value) will not be updated. So if an EOA calls contract A, and contract A performs a delegatecall to contract B, msg.sender will still be the EOA when contract B executes its code.

contract ContractA {
    uint private secretNumber;

    function updateSecret() public payable {
        secretNumber = 1337;
    }
}

contract ContractB {
    uint private secretNumber = 42;
    ContractA contractA = new ContractA();


    function callContractA() public payable {
        // B's storage is updated because of the delegatecall
        (bool success, bytes memory data) = address(contractA).delegatecall(abi.encodeWithSignature("updateSecret()"));
    }

    function getSecretNumber() external view returns(uint) {
        return secretNumber;
    }
}

In this example, ContractB has a private storage variable, secretNumber, equal to 42. By performing a delegatecall to ContractA, ContractA will update the secretNumber variable. This update is made in ContractB’s storage. So, following this call, the getSecretNumber() function will return 1337, not 42.

A classic use case for this type of call is proxy contracts. When a developer wants to update his contract, he has to deploy it again, and provide his users with the new address.

One solution is to create a proxy contract, in which all application information is stored, and this contract performs delegatecall to the real application. The developer communicates the proxy address to all his users.

If one day, the application needs to be updated, all that’s needed is to call a specific function in the proxy to update the application’s address. This update is transparent to users, since the proxy has not been modified.

Calls summary

Here’s a small table summarizing the different types of call.

Call from contract A to contract B	New EVM instance	Storage	msg.sender/msg.value	Blockchain state update
call	Yes	Contrat B	Updated	Possible
staticcall	Yes	Contrat B	Updated	Impossible
delegatecall	Yes	Contrat A	Not updated	Possible

Conclusion

This article has given us an overview of the EVM, Ethereum Virtual Machine. The virtual machine executes opcodes, within the limit of the gas sent by the user, since code execution has a cost.

To function properly, the EVM uses different memory areas to store temporary and persistent information.

Lastly, so that contracts can call each other, different calls are supported by the EVM.

These basics should be enough for us to take a serious look at smart contract vulnerabilities in future articles.

Ethereum

Mon, 10 Jul 2023 04:13:37 +0000

Unlike blockchains such as Bitcoin, which essentially allow Bitcoin cryptocurrency transactions to be sent, Ethereum also has something quite extraordinary: decentralized code execution.

Yes, decentralized. This means that we can write a program, code that is, and have it run not on one server, but on thousands of servers, or nodes. And the output of our program is also recorded in a decentralized way. I don’t know about you, but I think it’s incredible, and it really made me want to dig a little deeper into the subject.

So Ethereum is just another blockchain. There’s no shortage of blockchains today, but to date, Ethereum is the best-known and most widely used, at least in terms of blockchains that allow you to execute code. It does have its drawbacks, which other blockchains have addressed (albeit often to the detriment of other aspects), but that’s not really the point.

Let’s take a look at how Ethereum works, covering the notions of EOA accounts, contracts, states and transactions.

Ethereum 101

We discussed how blockchains work in general in the article Blockchain 101. Ethereum operates in a similar way, with the consensus mechanism being the Proof of Stake. Ethereum’s own cryptocurrency is Ether (or ETH). Like Bitcoin and all other blockchains, Ethers can be sent to other users via transactions. Each user has his own address.

What Ethereum brings is that, in addition to regular users who send transactions, it is possible to create small programs, smart contracts, which also live on the blockchain. They all have addresses, just like users, but they also have code, stored on the blockchain.

To distinguish these two types of accounts, we call classic human users EOA (Externally Owned Accounts), which we distinguish from contract accounts, which we’ll simply refer to as contracts.

EOA vs Contrats

Human-created accounts, or EOAs, are accounts with an address, a public key and a private key. They can initiate transactions by signing them, send Ethers and receive them. These transactions can be sent to other EOAs, allowing Ethers to be sent, but also to contracts.

Contracts also have an address, but no private key. They cannot initiate transactions. They can only react to transactions initiated by EOAs, or to messages sent by other contracts. Indeed, once called by an EOA, a contract can send messages to other contracts. The notion of message is discussed at the end of this article.

Data organization

Before we dive into why and how a contract account can execute code within the Ethereum ecosystem, let’s zoom in on the various data managed and used by Ethereum. Indeed, in this ecosystem, a global state of addresses must be kept up to date (with account balances, for example), the list of transactions must be stored and verifiable, the messages emitted in the various transactions must be accessible, and the permanent storage of each smart contract must, by definition, also be stored somewhere.

All this data is not stored in the blocks of the blockchain. Surprising as it may seem (at least to me at first sight), this information is stored in databases, outside the blocks, in the form of trees that follow a specific format: these are Merkle Patricia Tries, which enable a list of keys/values to be stored in an optimized way.

There’s no typo, it’s Trie, not Tree, in reference to the word Retrieve. We’ll probably see the Merkle Patricia Tries in detail in a dedicated article.

These data are therefore stored in the following trees:

State trie, or world state, which itself contains links to storage tries. Transactions tries
Receipt tries**

In blocks, only the hash of the root of each of these trees is stored.

It’s up to each client to know how to store the contents of the trees and manage queries based on the root node hash (not all clients use the same database solutions, by the way).

This enables lightweight devices (mobile, IoT) to synchronize quickly and easily with the blockchain without having to download huge volumes of data, so that they only know the root nodes hashes for each block. With this information, they can query full nodes, i.e. nodes that have stored the blockchain, and also all the data in the databases, to send them the few pieces of information they need to validate any given data.

Note that even an Ethereum full node only requires around 1TB of disk space. It’s accessible to everyone, which is why so many people participate in the decentralized network. There are also archive nodes. Unlike full nodes, which only synchronize with the last 128 blocks, archive nodes store the entire blockchain. For more information, please read this article.

Let’s see what these different data trees correspond to.

World State

Let’s start with the State Trie, or World State. We can point out that, while we were comparing a blockchain to a decentralized database, Ethereum is more complex and comprehensive than that. Instead, we could describe Ethereum as a decentralized state machine.

So, the general state of Ethereum is called World State. In this world state, there are all active user addresses (i.e. addresses present in at least one transaction), and each address has an associated account state.

Account State

The status of each account is therefore recorded in the world state containing the following 4 fields:

balance: the account’s Ether balance
nonce: A number that increments with each transaction for an EOA, and with each contract creation for a contract.
codeHash : A hash that can be used to retrieve the smart contract’s code (the hash of an empty character string for an EOA).
storageRoot: The root node hash of the Merkle Patricia Trie of account storage, or storage trie. It is used to retrieve the state of the contract, such as the value of permanently stored variables. This field is empty for an EOA account.

Each time a block on the blockchain is validated, all the transactions make changes to the world state, resulting in a new state.

In the following diagram, a block sends two transactions:

Address A sends 2 coins to address C. The balances (balance) of A and C will be updated, as will nonce of A (which increments with each transaction).
Address A sends 4 coins to address D. The balance of A will be updated, and address D, which does not yet exist in the world state, will be added, with a balance of 2, and an nonce equal to 0.

The fields in red are therefore those that are modified following the execution of the block’s transactions, leading to a new N+1 state.

Transactions

We now have a more precise overview of the account types that exist, and how they are stored within Ethereum. We’ve explained that blocks contain transactions which modify the state of the involved accounts, and therefore the general state, or world state. These transactions are actually recorded in a database, the Transactions Trie, in an orderly fashion.

A transaction contains several elements:

nonce: The nonce is specific to each account (stored for each address in the world state, if you’ve been following along), and is incremented for each new transaction.
gasPrice and gasLimit: Allow the user to define transaction fees
to: The destination address of the transaction
value`: The number of sent Eth
v,r,s`: The user’s signature
data: Allows you to send data to another account, or to create a contract

If you’re an observer, you’ll notice that a transaction must be signed. However, the only type of account that has a private key is the EOA. Contracts don’t have private keys. Therefore, they cannot initiate a transaction.

There are actually two types of transaction in Ethereum, those that allow you to send a message to another account, and those that allow you to create a contract.

Sending a message

In a transaction, account A sends a message to account B. The to destination address is that of account B, and the value and data fields can be used.

Sending Ether

To send Ether to the destination address, the desired amount will be set in value. When an account sends Ether to another account, only the value field is filled in. The destination account may be an EOA or a contract.

If the destination is a contract, the contract must have been designed to receive Ethers.

Sending data

The data field is mainly used to execute the code of a smart contract, when the transaction is intended for it. It’s also possible to send data to an EOA, and the recipient will process it as it sees fit.

When calling a contract function, the data field must be formatted as follows:

data: <Sélecteur de la fonction> <arguments>

The function’s selector is computed by hashing the function’s signature, keeping only the first 4 bytes.

For example, let’s imagine the following function:

function getItemValue(string calldata _itemName, uint256 _itemId) public returns(uint256 value) {
  // Function code
}

Function’s signature is:

getItemValue(string,uint256)

And it’s selector is:

bytes4(keccak256("getItemValue(string,uint256)"));
// Output:
0xc2e58fec

So data will be something like:

data: 0xc2e58fec<arguments>

We’ll see how the arguments are structured in a later article, but here’s an example for the getItemValue("pixis", 8) call:

0xc2e58fec                                                       # function selector
0000000000000000000000000000000000000000000000000000000000000040 # pointer to string "pixis"
0000000000000000000000000000000000000000000000000000000000000008 # 8
0000000000000000000000000000000000000000000000000000000000000005 # string length
7069786973000000000000000000000000000000000000000000000000000000 # string "pixis"

This type of message can therefore be sent from an EOA account transaction to a smart contract.

Note that it is also possible for a contract to call another contract’s function by sending the same message format. Everything will happen in the same transaction, since a contract cannot sign a new transaction. This type of call between contracts is a message call, a specific instruction of the Ethereum virtual machine. Only the message is sent, the destination contract is executed, and the result of this call is returned to the calling contract. We’ll look at these calls in more detail in future articles.

Contract creation

The second type of transaction enables an EOA account to create a new contract. To do this, the transaction is sent to the null address 0x00000..., and the data field is used.

This data field is divided into two parts:

The initialization bytecode, which is used to deploy the contract. It contains the contract constructor code and its arguments (if there is a constructor), as well as storage modifications if variables are declared. This code ends by returning the memory address of the *runtime bytecode and its size.
The runtime bytecode is the contract code, which includes all functions code.

Once this transaction has been processed, a new contract account is created. Its address is derived from the contract creator’s address and his nonce. In this way, each time a new contract is created by the same user, a different address is generated.

As we saw earlier, a new entry in the world state will be created for this address. The nonce will be 0, the contract balance will depend on the value field of the transaction that created it (0 by default), but the most important fields are :

codeHash: used to locate the account’s runtime bytecode, i.e. all the smart contract’s logic.
storageRoot: As a contract is always associated with a permanent storage, the account storage, this value is used to find this storage in order to read and modify all the variables used in the smart contract.

Receipts

The last Trie we haven’t talked about is the Receipts Trie. It is used to store information that is not required for the smart contracts to run properly, but which can be used by third-party applications, such as front-ends or clients.

This includes, for example, the status of the transaction (whether it failed or not), or the amount of gas used.

Furthermore, when a smart contract is executed, it can issue events.

contract MyContract {
  // "Transfer" event declaration
  Event Transfer(address to, uint value, uint tokenId);

  function transferTokens(address _to, uint _value, uint _tokenId) external {
    // Function code

    // "Transfer" event emitted
    emit Transfer(_to, _value, _tokenId);
  }
}

In this example, the Transfer event is emitted at the end of the transferToken function. This event will be added to the Receipts Trie.

Conclusion

These various elements give us a better understanding of how Ethereum works, what defines a smart contract, how a user can create a smart contract and how he can interact with it. This article, together with Blockchain 101, lays the foundations for explaining how Ethereum Virtual Machine (EVM) works. But that’s for the next article!

Blockchain 101

Mon, 03 Jul 2023 02:12:43 +0000

For several years now, I’ve been interested in a subject you’ve probably heard of: blockchains. I find it fascinating that a technology allows thousands of people to agree on so many subjects without the need for an intermediary. Decentralization is a subject that I believe has a lot of potential, and we’ll see in the long term whether this technology will endure or not. In any case, as it stands, it’s a hot topic! More recently, I’ve become interested in the Ethereum blockchain, smart contracts, and the security of smart contracts. We’re going to talk about all that here, here we go.

Before diving into the security of smart contracts, it’s important to recap some key concepts about blockchains. What is it, how does it work, who are the actors involved, we’ll look at all this in this introduction article. The idea is not to go into all the details, but to get an overview of how blockchains work in general. As the technical specifics vary greatly from one blockchain to another, we’ll cover them in due course in future articles.

Definition

There are hundreds of definitions for the term blockchain. What I think is important to understand is that it represents a decentralized registry (or database). There is no central entity deciding whether a transaction is valid or not, but rather thousands of people or machines working to verify and validate these transactions, all driven by mathematical rules and concepts.

In a nutshell, we can simplify a blockchain by imagining it as a huge Excel spreadsheet in which you can add rows one after the other. It is also possible to read the entire Excel file, from the moment it was created. However, it is not possible to modify a line that has already been written and validated. It’s append only.

Of course, this is a simplification, as blockchains such as Ethereum include, in addition to classic transactions, a virtual machine with its own storage space and so on. We’ll talk about this in the next article.

Transactions

What are these transactions? They are simply transfers of coins from one account to another. If Alice wants to send 1 coin to Bob, that’s a transaction.

A coin is the cryptocurrency of the blockchain. For the Bitcoin blockchain, it’s Bitcoin, for the Ethereum blockchain it’s Ether, for Solana it’s Sol, and so on.

To find out if Alice has enough coins, all you have to do is read the transaction history. The whole history. If one day she received 3 coins, spent 2 of them, then received 4, we can know, at the current time, that Alice has 3-2+4, i.e. 5 coins. She is then entitled to spend 1 coin, so everything’s fine.

Note that this is how Bitcoin works, but for other blockchains, the balance of each account is sometimes kept up to date (in the blockchain or not) to avoid having to recompute users’ balances for each transaction.

Here’s what a classic blockchain contains. A record of all users’ spending since the blockchain was created.

User

To be a blockchain user, you need to have a pair of asymmetric keys: a public key and a private key. The private key, obviously carefully stored by each user, is used to sign all transactions. This is how, when Alice claims to be sending 1 coin to some address, it is possible to verify that it was Alice who initiated the transaction. She has signed it with her private key, and anyone can check that this signature is valid with her public key.

This means that in a blockchain, we don’t know that the user is Alice. Rather, a user is defined by an address (derived from the public key). So when Alice wants to execute a transaction, from the blockchain’s point of view, her address is the source of the transaction.

Furthermore, to communicate with the blockchain, the user will use a client. This is nothing more than a program that knows how to generate transactions, communicate with the network and so on. The user could code everything himself, but that’s not practical. It’s a bit like using a web browser to go online. It’s more practical than writing code to make HTTP requests.

Validation

That’s all very well, but who validates these transactions? Who does the math to verify that Alice has at least 1 coin to send to someone? And checks that it’s really Alice who’s doing the transaction?

This is where the concepts of blocks and validators come in. For a blockchain to work properly, several people need to get to work validating transactions. They create so-called nodes, which will be able to broadcast themselves to the network and become part of it, retrieving past transactions and those awaiting validation. It’s a true peer-to-peer network. As soon as a user wants to send a transaction (1), the client he is using to send his transaction will notify another node (via NewPooledTransactionHashes) that a new transaction has been sent (2). The transaction will be verified (signature verification, funds available, etc.) (3), but it is not yet validated. It will join the waiting list of transactions that have been sent but not yet validated, called the mempool. This node will also notify other nodes by broadcasting this transaction (4), and these new nodes will do the verification work themselves (6) and add this transaction to their mempool, and so on.

So there’s a whole bunch of transactions waiting to be validated, and that’s where the magic of the blockchain comes in. Transactions have to be validated, and all the nodes in the network will need to agree on which transactions should be validated, and the order in which they should be validated.

Each node then creates a block, the size of which is limited (and differs from one blockchain to another) by selecting pending transactions in the mempool. Once this block has been created, all the nodes compete to make its block the new reference block. The winner’s block becomes the last block in the chain. It is added to the previously validated blocks, the underlying transactions are no longer in the mempool, since they have been validated, and all nodes must therefore rebuild a new block with the transactions that have not yet been validated to try, once again, to win this competition.

Consensus

This “competition” we’re talking about is the consensus mechanism, i.e. a way of getting everyone to agree on the next block. There are many different consensus mechanisms.

The Proof of Work (PoW) is a consensus mechanism that requires each node to make a huge number of calculations to find a solution to a specific problem.

In simple terms, it’s as if you were asked to supply a string such that md5(block + string) begins with ten times the number 0. There’s really no right or wrong way to do this.

You can simply generate completely random strings, compute the md5 hash, until you find, by chance, an entry that satisfies the condition. And at some point, in a completely random fashion, someone may test :

echo -n '[bloc data]aa33bdsk' | md5sum

# Output:
000000000035d3695b3a133766f60d42

By being the first to find this solution to the problem, their block will be added to the existing blockchain, and therefore the transactions they took from the mempool will be validated.

Proof of Stake (PoS) avoids the need for all all nodes to perform calculations. Instead, each node must set aside cryptocurrencies from the blockchain (staking). Each node prepares its block, and at regular intervals, an algorithm randomly selects a node from those that have staked cryptocurrencies. The selected node’s block will be validated, and we move on to the next block. If the node does not respect the rules or tries to cheat (by modifying transactions or creating a block that is too big, for example), his stake will be confiscated. You gotta play by the rules.

There are many other consensus mechanisms, but you get the idea. The goal is for a random node to validate a block regularly, but it should not be possible for the same node to validate all blocks. Everyone is in competition.

Incentives

Rest assured, the individuals behind these nodes are not blockchain enthusiasts working for free. Every job deserves a salary, and that applies to the blockchain as well. The people who are part of the network, verifying and validating transactions, earn rewards.

To send a transaction on the network, users must also send a small amount called transaction fees (known as gas in Ethereum). Thus, when someone validates a block, they will collect the fees from the transactions they have validated. It becomes clear then that as a user, if we want to ensure that our transaction doesn’t stay indefinitely in the mempool, we need to pay sufficient transaction fees to be in the average range or even at the higher end if we want to be prioritized.

Furthermore, with each validated block, a small amount of the cryptocurrency is created from scratch and sent to the validator. This increases the total circulating supply of the coin.

Conclusion

These paragraphs hopefully clarify the overall concept of a blockchain and serve as an introduction to the next articles that focus on the Ethereum blockchain, specifically the Ethereum Virtual Machine, which enables the execution of smart contracts, and the security challenges associated with this decentralized code execution. See you soon!

Kerberos Delegation

Sat, 18 Apr 2020 12:17:22 +0000

Within an Active Directory, services can be used by users. Sometimes these services need to contact others, on behalf of the user, like a web service might need to contact a file server. In order to allow a service to access another service on behalf of the user, a solution has been implemented (introduced in Windows Server 2000) to meet this need : Kerberos Delegation.

Delegation principle

In order to understand what is Kerberos Delegation, let’s take a concrete example. A web server with a nice interface allows a user to access his personal folder, hosted on a file server. We are in the following situation :

The web server is front-end, and it’s this web server that will fetch the information instead of the user on the file server in order to display the content of a file, for example.

However, the web server does not know what belongs to the user on the file server. It is not his role to unpack the user’s PAC to make a specific demand to the file server. This is where the delegation comes in. This mechanism allows the web server to impersonate the user, and to authenticate on the user’s behalf to the file server. Thus, from the file server’s point of view, it is the user who makes the request. The file server will be able to read and check user’s rightsr, then send back the information to which this account has access. This is how the web server can then display this information in a nice interface back to the user.

Constrained & Unconstrained Delegation

The ability to relay credentials can be given to an account with at least one SPN attribute set. It could be a computer account or a service account.

Today, there are three ways to authorize a computer or service account to impersonate a user in order to communicate with one or more other service(s) : Unconstrained Delegation, Constrained Delegation and Resource Based Constrained Delegation.

Unconstrained Delegation

With Unconstrained Delegation, the server or the service account that is granted this right is able to impersonate a user to authenticate to any services on any host.

Here is an example, in my lab, of a machine that is in Unconstrained Delegation :

It is historically the only choice there was when the delegation principle was introduced. But for obvious security reason, Constrained Delegation has been added.

Constrained Delegation

If a computer or a service account has the Constrained Delegation flag set, a list of authorized services shall be associated to this flag. For example, in previous example (web server and file server), the web server will have the Constrained Delegation flag indicating that this account can only impersonate some users against CIFS service hosted by SERVER01, the file server.

This list of allowed SPN is set on the relaying account, the computer account hosting the web server..

In other words, the Domain Controller will read the SPN list on this account, and will decide: “This account is allowed to impersonate a user, so he can authenticate against one of these services on behalf of the user”.

In my lab, the web server is WEB-SERVER-01 and the one with file sharing is WEB-SERVER-02. So here is what the list of services for which WEB-SERVER-01 can pretend to be the user looks like :

Resource Based Constrained Delegation - RBCD

Finally, we have the Resource Based Constrained Delegation (RBCD) case. Introduced with Windows Server 2012, this solution allows to overcome some problems related to the Constrained Delegation (Responsibility, inter-domains delegation, …). Without going into too much details, the delegation responsibility is moved. Whereas in Constrained Delegation, it’s the relaying server that holds the list of allowed target services, in the case of Resource Based Constrained Delegation, it’s the resources (or services) that have a list of accounts they trust for delegation. Thus, the diagram is as follows :

The responsibility is shifted, it’s at the level of the resource that receives the delegated authentications that the information of whether or not the delegation is accepted is found.

In other words, it’s the end resource that says “I allow this list of account […] to authenticate to me on behalf of someone else”.

Technical details

Now that the high-level behavior is understood (at least I hope so), let’s go into a little more detail about this process. Concretely, how can a machine or an account can pretend to be a user when using a resource ? That’s what we are going to see now. Details between every different techniques are relatively different, that’s why each of them will be explained separately. Stay close on your seat, it’s gonna get dirty.

Unconstrained Delegation

As seen before, in this case, the server or the service account can authenticate on behalf of the user to any other services. For this to be possible, two prerequisites are required :

The first one is that the account that wants to delegate an authentication has the TRUSTED_FOR_DELEGATION flag in his UAC - User Account Control flags. In order to set this flag, you need to have the SeEnableDelegationPrivilege right, which is usually only available for domain administrators. Here is how the flag is set on the account (machine or service account):

Once set, we can list UAC flags and check that TRUSTED_FOR_DELEGATION has been set.

The second one is that the user account which will be relayed is effectively “relayable”. To disable relaying capabilities on an account, NOT_DELEGATED flag is set. By default, no account on the AD has this flag set, so they are all “relayable”.

Concretely, during exchanges with the Domain Controller as described in the Kerberos in Active Directory article, when the user asks for a TGS (KRB_TGS_REQ), he will specify the SPN of the service he wants to use. It is at this point that the Domain Controller will look for the two prerequisites :

Is the TRUSTED_FOR_DELEGATION flag set in the attributes of the account associated to the SPN.
Is the NOT_DELEGATED flag not set for the requesting user.

If both prerequisites are met, then the Domain Controller will respond to the user with a KRB_TGS_REP containing standard information, but it will also contains a copy of the user’s TGT in his response, and a new associated session key.

Once in possession of these elements, the user will continue the classic process by sending a request to the service (KRB_AP_REQ) by sending the TGS and an authenticator. The service will be able to decrypt the content of the TGS, verify the user’s identity by decrypting the authenticator, but above all it will be able to retrieve the copy of the TGT and the associated session key, in order to pretend to be the user at the Domain Controller.

Now in possession of a copy of the user’s TGT and a valid session key, the service can authenticate to any other service on the user’s behalf by requesting the TGS from the Domain Controller, by providing this TGT and by encrypting an authenticator with the session key. It’s the Unconstrained Delegation principle.

Here is a summary diagram :

Constrained Delegation

For the Constrained Delegation, a list of SPN or of authorized accounts will be provided to indicate the services/accounts accepted for the delegation. Therefore, the process is not the same. The service involved will not be in possession of the user’s TGT, otherwise there is no way to control service’s authentication. A different mechanism is used.

Let’s consider the case where the user authenticate to Service A and then this Service A has to impersonate the user to access Resource B.

The user makes a TGS request, then sends it to Service A. Since this service needs to access Resource B as the user, it will request a TGS to the Domain Controller on behalf of the user. This request is governed by the S4U2Proxy extension. To tell the Domain Controller it wants to authenticate on behalf of someone else, two attributes will be defined in the KRB_TGS_REQ ticket request:

The field additional-tickets, usually empty, must contain the user’s TGS (given that the NOT_DELEGATED flag is not set for the requesting user. If that was the case, the user’s TGS would not be forwardable, and the Domain Controller would not accept it in the rest of the process)
The cname-in-addl-tkt flag, which should be set to indicate to the Domain Controller that it should not use the server information, but the ticket information in additional-tickets, i.e. the information of the user the server wants to pretend to be.

It is during this request that the Domain Controller, upon seeing this information, will verify that Service A has the right to authenticate to Resource B on behalf of the user.

Constrained Delegation - Classic

In the classic Constrained Delegation case (when delegation information is located in Service A), this information is found in the msDS-AllowedToDelegateTo attribute of the requesting account, thus of Service A. This attribute specifies the list of authorized SPN for the delegation.

For example here the msDS-AllowedToDelegateTo attribute will contain cifs/WEB-SERVER-02.

If the targeted SPN is present, then the Domain Controller sends back a valid TGS, with the name of the user, for the requested service. Here is a summary diagram :

Constrained Delegation - Resource Based

This time, the Domain Controller will look at the attributes of Resource B (instead of Service A). It will check that the account associated with Service A is present in the msDS-AllowedToActOnBehalfOfOtherIdentity attribute of the account linked to Resource B.

In order to set up RBCD on a resource, we need to use powershell.

WEB-SERVER-01 is added to WEB-SERVER-02 trusting list, so msDS-AllowedToActOnBehalfOfOtherIdentity is updated.

Then, when a service wants to access a resource, here’s how it works:

As the diagram shows, the technical functioning is similar to classic constrained delegation, however the responsibilities of each entity is radically different.

S4U2Self

If you are still there and you are not lost, you should have noticed that we haven’t touched on the notion of protocol transition in this article. Indeed, in the explanation of constrained delegation, we assumed that Service A had a service ticket coming from USER, which was added in the additional-tickets field of the TGS request (S4U2Proxy). But sometimes the user may authenticate to the server in other ways than the Kerberos protocol (e.g. via NTLM, or even a Web form). In this case, the server is not in possession of the TGS sent by the user. Thus, Service A cannot fill the additional-tickets field as it did in the case described above.

That is why there is an extra step, possible through the S4U2Self extension, that Service A must perform. This step allows it to obtain a TGS for a user arbitrarily chosen. To do this, it makes a classic TGS request (KRB_TGS_REQ) except that instead of putting his own name in the PA-FOR-USER block (present in the pre-authentication part), it puts the name of a user it chooses.

This ability to manage protocol transition is accepted by the Domain Controller only if it has been explicitly granted to the service account wishing to manage this delegation. It is in delegation management tab that an administrator can choose a constrained delegation using Kerberos only, therefore without protocol transition, or using any protocol, thus allowing protocol transition.

In the first case, no specific flag is set on the account, and the service can only relay kerberos authentication. It cannot use the S4U2Self extension to create a ticket out of nowhere.

In the second case, the TRUSTED_TO_AUTHENTICATE_FOR_DELEGATION flag is set. If this is the case, then the service with this flag can pretend to be anyone when accessing services in its list via the S4U2Self extension.

This diagram is getting a little bit more complicated, but I hope it remains relatively clear.

If such an account is compromised, then all services to which that account is entitled to authenticate via delegation will also be compromised, since the attacker can create service tickets on behalf of arbitrary users, such as administrators.

Conclusion

I was thinking of doing an article that would describe kerberos delegation as well as associated attacks, however the explanations are much more dense than expected, so this article remains devoted to explanation. Associated attacks will be presented in other articles.

To sum up, there are three types of delegation:

Unconstrained delegation: In this case, the client sends a copy of his TGT to a service, and that service uses it to impersonate the client to any other service. Only an administrator can set this option on an account.
Constrained delegation: A list of resources is set on the service that wishes to delegate authentication. If protocol transition is allowed, then the service can pretend to be anyone when accessing resources in its list. In any case, only an administrator can set this option.
Resource-based Constraint Delegation: The final resource has a list of trusted accounts. All accounts in this list can delegate authentication when accessing the resource. Resources can modify this list as they wish, they don’t need an administrator to update it.

If you have any questions or suggestions, do not hesitate, I’m all ears.

Resources

Big thanks to them for their clear explanations.

NTLM Relay

Wed, 01 Apr 2020 10:11:52 +0000

NTLM relay is a technique of standing between a client and a server to perform actions on the server while impersonating the client. It can be very powerful and can be used to take control of an Active Directory domain from a black box context (no credentials). The purpose of this article is to explain NTLM relay, and to present its limits.

Preliminary

This article is not meant to be a tutorial to be followed in order to carry out a successful attack, but it will allow the reader to understand in detail the technical details of this attack, its limitations, and can be a basis to start developing his own tools, or understand how current tools work.

In addition, and to avoid confusion, here are some reminders:

NT Hash and LM Hash are hashed versions of user passwords. LM hashes are totally obsolete, and will not be mentioned in this article. NT hash is commonly called, wrongly in my opinion, “NTLM hash”. This designation is confusing with the protocol name, NTLM. Thus, when we talk about the user’s password hash, we will refer to it as NT hash.
NTLM is therefore the name of the authentication protocol. It also exists in version 2. In this article, if the version affects the explanation, then NTLMv1 and NTLMv2 will be the terms used. Otherwise, the term NTLM will be used to group all versions of the protocol.
NTLMv1 Response and NTLMv2 Response will be the terminology used to refer to the challenge response sent by the client, for versions 1 and 2 of the NTLM protocol.
Net-NTLMv1 and Net-NTLMv2 are pseudo-neo-terminologies used when the NT hash is called NTLM hash in order to distinguish the NTLM hash from the protocol. Since we do not use the NTLM hash terminology, these two terminologies will not be used.
Net-NTLMv1 Hash and Net-NTLMv2 Hash are also terminologies to avoid confusion, but will also not be used in this article.

Introduction

NTLM relay relies, as its name implies, on NTLM authentication. The basics of NTLM have been presented in pass-the-hash article. I invite you to read at least the part about NTLM protocol and local and remote authentication.

As a reminder, NTLM protocol is used to authenticate a client to a server. What we call client and server are the two parts of the exchange. The client is the one that wishes to authenticate itself, and the server is the one that validates this authentication.

This authentication takes place in 3 steps:

First the client tells the server that it wants to authenticate.
The server then responds with a challenge which is nothing more than a random sequence of characters.
The client encrypts this challenge with its secret, and sends the result back to the server. This is its response.

This process is called challenge/response.

The advantage of this exchange is that the user’s secret never passes through the network. This is known as Zero-knowledge proof.

NTLM Relay

With this information, we can easily imagine the following scenario: An attacker manages to be in a man-in-the-middle position between a client and a server, and simply relays information from one to the other.

The man-in-the-middle position means that from the client’s point of view, the attacker’s machine is the server to which he wants to authenticate, and from the server’s point of view, the attacker is a client like any other who wants to authenticate.

Except that the attacker does not “just” want to authenticate to the server. He wishes to do so by pretending to be the client. However, he does not know the secret of the client, and even if he listens to the conversations, as this secret is never transmitted over the network (zero-knowledge proof), the attacker is not able to extract any secret. So, how does it work?

Message Relaying

During NTLM authentication, a client can prove to a server its identity by encrypting with its password some piece of information provided by the server. So the only thing the attacker has to do is to let the client do its work, and passing the messages from the client to the server, and the replies from the server to the client.

All the client has to send to the server, the attacker will receive it, and he will send the messages back to the real server, and all the messages that the server sends to the client, the attacker will also receive them, and he will forward them to the client, as is.

And it’s all working out! Indeed, from the client’s point of view, on the left part on the diagram, an NTLM authentication takes place between the attacker and him, with all the necessary bricks. The client sends a negotiate request in its first message, to which the attacker replies with a challenge. Upon receiving this challenge, the client builds its response using its secret, and finally sends the last authentication message containing the encrypted challenge.

Ok, that’s great but the attacker cannot do anything with this exchange. Fortunately, there is the right side of the diagram. Indeed, from the server’s point of view, the attacker is a client like any other. He sent a first message to ask for authentication, and the server responded with a challenge. As the attacker sent this same challenge to the real client, the real client encrypted this challenge with its secret, and responded with a valid response. The attacker can therefore send this valid response to the server.

This is where the interest of this attack lies. From the server’s point of view, the attacker has authenticated himself using the victim’s secret, but in a transparent way for the server. It has no idea that the attacker was replaying his messages to the client in order to get the client to give him the right answers.

So, from the server’s point of view, this is what happened:

At the end of these exchanges, the attacker is authenticated on the server with the client’s credentials.

Net-NTLMv1 and Net-NTLMv2

For information, it is this valid response relayed by the attacker in message 3, the encrypted challenge, that is commonly called Net-NTLMv1 hash or Net-NTLMv2 hash. But in this article, it will be called NTLMv1 response or NTLMv2 response, as indicated in the preliminary paragraph.

To be exact, this is not exactly an encrypted version of the challenge, but a hash that uses the client’s secret. It is HMAC_MD5 function which is used for NTLMv2 for example. This type of hash can only be broken by brute force. The cryptography associated with computation of the NTLMv1 hash is obsolete, and the NT hash that was used to create the hash can be retrieved very quickly. For NTLMv2, on the other hand, it takes much longer. It is therefore preferable and advisable not to allow NTLMv1 authentication on a production network.

In practice

As an example, I set up a small lab with several machines. There is DESKTOP01 client with IP address 192.168.56.221 and WEB01 server with IP address 192.168.56.211. My host is the attacker, with IP address 192.168.56.1. So we are in the following situation:

The attacker has therefore managed to put himself man-in-the-middle position. There are different techniques to achieve this, whether through abuse of default IPv6 configurations in a Windows environment, or through LLMNR and NBT-NS protocols. Either way, the attacker makes the client think that he is the server. Thus, when the client tries to authenticate itself, it is with the attacker that it will perform this operation.

The tool I used to perform this attack is ntlmrelayx from impacket. This tool is presented in details in this article by Agsolino, impacket (almighty) developer.

ntlmrelayx.py -t 192.168.56.211

The tool creates different servers, including an SMB server for this example. If it receives a connection on this server, it will relay it to the provided target, which is 192.168.56.211 in this example.

From a network point of view, here is a capture of the exchange, with the attacker relaying the information to the target.

In green are the exchanges between DESKTOP01 client and the attacker, and in red are the exchanges between the attacker and WEB01 server. We can clearly see the 3 NTLM messages between DESKTOP01 and the attacker, and between the attacker and WEB01 server.

And to understand the notion of relay, we can verify that when WEB01 sends a challenge to the attacker, the attacker sends back exactly the same thing to DESKTOP01.

Here is the challenge sent by WEB01 to the attacker :

When the attacker receives this challenge, he sends it to DESKTOP01 without any modification. In this example, the challenge is b6515172c37197b0.

The client will then compute the response using his secret, as we have seen in the previous paragraphs, and he will send his response alongside with his username (jsnow), his hostname (DESKTOP01), and in this example he indicates that he is a domain user, so he provides the domain name (ADSEC).

The attacker who gets all that doesn’t ask questions. He sends the exact same information to the server. So he pretends to be jsnow on DESKTOP01 and part of ADSEC domain, and he also sends the response that has been computed by the client, called NTLM Response in these screenshots. We call this response NTLMv2 hash.

We can see that the attacker was only relaying stuff. He just relayed the information from the client to the server and vice versa, except that in the end, the server thinks that the attacker is successfully authenticated, and the attacker can then perform actions on the server on behalf of ADSEC\jsnow.

Authentication vs Session

Now that we have understood the basic principle of NTLM relay, the question that arises is how, concretely, can we perform actions on a server after relaying NTLM authentication? By the way, what do we mean by “actions”? What is it possible to do?

To answer this question, we must first clarify one fundamental thing. When a client authenticates to a server to do something, we must distinguish two things:

Authentication, allowing the server to verify that the client is who he claims to be.
The session, during which the client will be able to perform actions.

Thus, if the client has authenticated correctly, it will then be able to access the resources offered by the server, such as network shares, access to an LDAP directory, an HTTP server or a SQL database. This list is obviously not exhaustive.

To manage these two steps, the protocol that is used must be able to encapsulate the authentication, thus the exchange of NTLM messages.

Of course, if all the protocols were to integrate NTLM technical details, it would quickly become a holy mess. That’s why Microsoft provides an interface that can be relied on to handle authentication, and packages have been specially developed to handle different types of authentication.

SSPI & NTLMSSP

The SSPI interface, or Security Support Provider Interface, is an interface proposed by Microsoft to standardize authentication, regardless of the type of authentication used. Different packages can connect to this interface to handle different types of authentication.

In our case, it is the NTLMSSP package (NTLM Security Support Provider) that interests us, but there is also a package for Kerberos authentication, for example.

Without going into details, the SSPI interface provides several functions, including AcquireCredentialsHandle, InitializeSecurityContext and AcceptSecurityContext.

During NTLM authentication, both the client and the server will use these functions. The steps are only briefly described here.

The client calls AcquireCredentialsHandle in order to gain indirect access to the user credentials.
The client then calls InitializeSecurityContext, a function which, when called for the first time, will create a message of type 1, thus of type NEGOTIATE. We know this because we’re interested in NTLM, but for a programmer, it doesn’t matter what this message is. All that matters is to send it to the server.
The server, when receiving the message, calls the AcceptSecurityContext function. This function will then create the type 2 message, the CHALLENGE.
When receiving this message, the client will call InitializeSecurityContext again, but this time passing the CHALLENGE as an argument. The NTLMSSP package takes care of everything to compute the response by encrypting the challenge, and will produce the last AUTHENTICATE message.
Upon receiving this last message, the server also calls AcceptSecurityContext again, and the authentication verification will be performed automatically.

The reason these steps are explained is to show that in reality, from the client or server point of view, the structure of the 3 messages that are exchanged does not matter. We, with our knowledge of the NTLM protocol, know what these messages correspond to, but both the client and the server don’t care. These messages are described in the Microsoft documentation as opaque tokens.

This means that these 5 steps are completely independent of client’s type or server’s type. They work regardless of the protocol used as long as the protocol has something in place to allow this opaque structure to be exchanged in one way or another from the client to the server.

So the protocols have adapted to find a way to put an NTLMSSP, Kerberos, or other authentication structure into a specific field, and if the client or server sees that there is data in that field, it just passes it to InitializeSecurityContext or AcceptSecurityContext.

This point is quite important, since it clearly shows that the application layer (HTTP, SMB, SQL, …) is completely independent from the authentication layer (NTLM, Kerberos, …). Therefore, security measures are both needed for the authentication layer and for the application layer.

For a better understanding, we will see the two examples of application protocols SMB and HTTP. It’s quite easy to find documentation for other protocols. It’s always the same principle.

Integration with HTTP

This is what a basic HTTP request looks like.

GET /index.html HTTP/1.1
Host: beta.hackndo.com
User-Agent: Mozilla/5.0
Accept: text/html
Accept-Language: fr

The mandatory elements in this example are the HTTP verb (GET), the path to the requested page (/index.html), the protocol version (HTTP/1.1), or the Host header (Host: beta.hackndo.com).

But it is quite possible to add other arbitrary headers. At best, the remote server is aware that these headers will be present, and it will know how to handle them, and at worst it will ignore them. This allows you to have the same request with some additional information.

GET /index.html HTTP/1.1
Host: beta.hackndo.com
User-Agent: Mozilla/5.0
Accept: text/html
Accept-Language: en
X-Name: pixis
Favorite-Food: Beer 'coz yes, beer is food

It is this feature that is used to be able to transfer NTLM messages from the client to the server. It has been decided that the client sends its messages in a header called Authorization and the server in a header called WWW-Authenticate. If a client attempts to access a web site requiring authentication, the server will respond by adding the WWW-Authenticate header, and highlighting the different authentication mechanisms it supports. For NTLM, it will simply say NTLM.

The client, knowing that NTLM authentication is required, will send the first message in the Authorization header, encoded in base 64 because the message does not only contain printable characters. The server will respond with a challenge in the WWW-Authenticate header, the client will compute the response and will send it in Authorization. If authentication is successful, the server will usually return a 200 return code indicating that everything went well.

> GET /index.html HTTP/1.1
> Host: beta.hackndo.com
> User-Agent: Mozilla/5.0
> Accept: text/html
> Accept-Language: en

  < HTTP/1.1 401 Unauthorized
  < WWW-Authenticate: NTLM
  < Content type: text/html
  < Content-Length: 0

> GET /index.html HTTP/1.1
> Host: beta.hackndo.com
> User-Agent: Mozilla/5.0
> Accept: text/html
> Accept-Language: en
=> Authorization: NTLM <NEGOTIATE in base 64>

  < HTTP/1.1 401 Unauthorized
  => WWW-Authenticate: NTLM <CHALLENGE in base 64>
  < Content type: text/html
  < Content-Length: 0

> GET /index.html HTTP/1.1
> Host: beta.hackndo.com
> User-Agent: Mozilla/5.0
> Accept: text/html
> Accept-Language: en
=> Authorization: NTLM <RESPONSE in base 64>

  < HTTP/1,200 OKAY.
  < WWW-Authenticate: NTLM
  < Content type: text/html
  < Content-Length: 0
  < Connection: close

As long as the TCP session is open, authentication will be effective. As soon as the session closes, however, the server will no longer have the client’s security context, and a new authentication will have to take place. This can often happen, and thanks to Microsoft’s SSO (Single Sign On) mechanisms, it is often transparent to the user.

Integration with SMB

Let’s take another example frequently encountered in a company network. It is SMB protocol, used to access network shares, but not only.

SMB protocol works by using commands. They are documented by Microsoft, and there are many of them. For example, there are SMB_COM_OPEN, SMB_COM_CLOSE or SMB_COM_READ, commands to open, close or read a file.

SMB also has a command dedicated to configuring an SMB session, and this command is SMB_COM_SESSION_SETUP_ANDX. Two fields are dedicated to the contents of the NTLM messages in this command.

LM/LMv2 Authentication: OEMPassword
NTLM/NTLMv2 authentication: UnicodePassword

What is important to remember is that there is a specific SMB command with an allocated field for NTLM messages.

Here is an example of an SMB packet containing the response of a server to an authentication.

These two examples show that the content of NTLM messages is protocol-independent. It can be included in any protocol that supports it.

It is then very important to clearly distinguish the authentication part, i.e. the NTLM exchanges, from the application part, or the session part, which is the continuation of the exchanges via the protocol used once the client is authenticated, like browsing a website via HTTP or accessing files on a network share if we use SMB.

As this information is independent, it means that a man-in-the-middle may very well receive authentication via HTTP, for example, and relay it to a server but using SMB. This is called cross-protocol relay.

With all these aspects in mind, the following chapters will highlight the various weaknesses that exist or have existed, and the security mechanisms that come into play to address them.

Session signing

Principle

A signature is an authenticity verification method, and it ensures that the item has not been tampered with between sending and receiving. For example, if the user jdoe sends the text I love hackndo, and digitally signs this document, then anyone who receives this document and his signature can verify that it was jdoe who edited it, and can be assured that he wrote this sentence, and not another, since the signature guarantees that the document has not been modified.

The signature principle can be applied to any exchange, as long as the protocol supports it. This is for example the case of SMB, LDAP and even HTTP. In practice, the signing of HTTP messages is rarely implemented.

But then, what’s the point of signing packages? Well, as discussed earlier, session and authentication are two separate steps when a client wants to use a service. Since an attacker can be in a man-in-the-middle position and relay authentication messages, he can impersonate the client when talking with the server.

This is where signing comes into play. Even if the attacker has managed to authenticate to the server as the client, he will not be able to sign packets, regardless of authentication. Indeed, in order to be able to sign a packet, one must know the secret of the client.

In NTLM relay, the attacker wants to pretend to be a client, but he has no knowledge of his secret. He is therefore unable to sign anything on behalf of the client. Since he can’t sign any packet, the server receiving packets will either see that the signature is not present, or that it doesn’t exist, and will reject the attacker’s request.

So you understand that if packets must necessarily be signed after authentication, then the attacker can no longer operate, since he has no knowledge of the client’s secret. So the attack will fail. This is a very effective measure to protect against NTLM relay.

That’s all very well, but how do the client and the server agree on whether or not to sign packets? Well that’s a very good question. Yes, I know, I’m the one asking it, but that doesn’t make it irrelevant.

There are two things that come into play here.

The first one is to indicate if signing is supported. This is done during NTLM negotiation.
The second one allows to indicate if signing will be required, optional or disabled. This is a setting that is done at the client and server level.

NTLM Negotiation

This negotiation allows to know if the client and/or the server support signing (among other things), and is done during the NTLM exchange. So I lied to you a bit earlier, authentication and session are not completely independent. (By the way, I said that since it was independent, you could change protocol when relaying, but there are limits, we will see them in the chapter on MIC in NTLM authentication).

In fact, in NTLM messages, there is other information other than a challenge and the response that are exchanged. There are also negotiation flags, or Negotiate Flags. These flags indicate what the sending entity supports.

There are several flags, but the one of interest here is NEGOTIATE_SIGN.

When this flag is set to 1 by the client, it means that the client supports signing. Be careful, it does not mean that he will necessarily sign his packets. Just that he’s capable of it.

Similarly when the server replies, if it supports signing then the flag will also be set to 1.

This negotiation thus allows each of the two parties, client and server, to indicate to the other if it is able sign packets. For some protocols, even if the client and the server support signing, this does not necessarily mean that the packets will be signed.

Implementation

Now that we’ve seen how both parties indicate to the other their ability to sign packets, they have to agree on it. This time, this decision is made according to the protocol. So it will be decided differently for SMBv1, for SMBv2, or LDAP. But the idea remains the same.

Depending on the protocol, there are usually 2 or even 3 options that can be set to decide wether signing will be enforced, or not. The 3 options are :

Disabled: This means that signing is not managed.
Enabled: This option indicates that the machine can handle signing if need be, but it does not require signing.
Mandatory: This finally indicates that signing is not only supported, but that packets must be signed in order for the session to continue.

We will see here the example of two protocols, SMB and LDAP.

SMB

Signature matrix

A matrix is provided in Microsoft documentation to determine whether or not SMB packets are signed based on client-side and server-side settings. I’ve reported it in this table. Note however that for SMBv2 and higher, signing is necessarily handled, the Disabled parameter no longer exists.

There is a difference when client and server have the Enabled setting. In SMBv1, the default setting for servers was Disabled. Thus, all SMB traffic between clients and servers was not signed by default. This avoided overloading the servers by preventing them from computing signatures each time an SMB packet was sent. As the Disabled status no longer exists for SMBv2, and servers are now Enabled by default, in order to keep this load saving, the behavior between two Enable entites has been modified, and signing is no longer required in this case. The client and/or the server must necessarily require the signature for SMB packets to be signed.

Settings

In order to change the default signing settings on a server, the EnableSecuritySignature and RequireSecuritySignature keys must be changed in registry hive HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanServer\Parameters.

This screenshot was taken on a domain controller. By default, domain controllers require SMB signing when a client authenticates to them. Indeed, the GPO applied to domain controllers contains this entry:

On the other hand, we can see on this capture that above this setting, the same parameter applied to Microsoft network client is not applied. So when the domain controller acts as an SMB server, SMB signing is required, but if a connection comes from the domain controller to a server, SMB signing is not required.

Setup

Now that we know where SMB signing is configured, we can see this parameter applied during an communication. It is done just before authentication. In fact, when a client connects to the SMB server, the steps are as follows:

Negotiation of the SMB version and signing requirements
Authentication
SMB session with negotiated parameters

Here is an example of SMB signing negotiation:

We see a response from a server indicating that it has the Enable parameter, but that it does not require signing.

To summarize, here is how a negotiation / authentication / session takes place :

In the negotiation phase, both parties indicate their requirements: Is signing required for one of them?
In the authentication phase, both parties indicate what they support. Are they capable of signing?
In the session phase, if the capabilities and the requirements are compatible, the session is carried out applying what has been negotiated.

For example, if DESKTOP01 client wants to communicate with DC01 domain controller, DESKTOP01 indicates that it does not require signing, but that that he can handle it, if needed.

DC01 indicates that not only he supports signing, but that he requires it.

During negotiation phase, the client and server set the NEGOTIATE_SIGN flag to 1 since they both support signing.

Once this authentication is completed, the session continues, and the SMB exchanges are effectively signed.

LDAP

Signing matrix

For LDAP, there are also three levels:

Disabled: This means that packet signing is not supported.
Negotiated Signing: This option indicates that the machine can handle signing, and if the machine it is communicating with also handles it, then they will be signed.
Required: This finally indicates that signing is not only supported, but that packets must be signed in order for the session to continue.

As you can read, the intermediate level, Negotiated Signing differs from the SMBv2 case, because this time, if the client and the server are able to sign packets, then they will. Whereas for SMBv2, packets were only signed if it was a requirement for at least one entity.

So for LDAP we have a matrix similar to SMBv1, except for the default behaviors.

The difference with SMB is that in Active Directory domain, all hosts have Negotiated Signing setting. The domain controller doesn’t require signing.

Settings

For the domain controller, the ldapserverintegrity registry key is in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS\Parameters hive and can be 0, 1 or 2 depending on the level. It is set to 1 on the domain controller by default.

For the clients, this registry key is located in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ldap

It is also set to 1 for clients. Since all clients and domain controllers have Negotiated Signing, all LDAP packets are signed by default.

Setup

Unlike SMB, there is no flag in LDAP that indicates whether packets will be signed or not. Instead, LDAP uses flags set in NTLM negotiation. There is no need for more information. In the case where both client and server support LDAP signing, then the NEGOTIATE_SIGN flag will be set and the packets will be signed.

If one party requires signing, and the other party does not support it, then the session simply won’t start. The party requiring signing will ignore the unsigned packets.

So now we then understand that, contrary to SMB, if we are between a client and a server and we want to relay an authentication to the server using LDAP, we need two things :

The server must not require packet signing, which is the case for all machines by default
The client must not set the NEGOTIATE_SIGN flag to 1. If he does, then the signature will be expected by the server, and since we don’t know the client’s secret, we won’t be able to sign our crafted LDAP packets.

Regarding requirement 2, sometimes clients don’t set this flag, but unfortunately, the Windows SMB client sets it! By default, it is not possible to relay SMB authentication to LDAP.

So why not just change the NEGOTIATE_FLAG flag and set it to 0? Well… NTLM messages are also signed. This is what we will see in the next paragraph.

Authentication signing (MIC)

We saw how a session could be protected against a middle man attacker. Now, to understand the interest of this chapter, let’s look at a specific case.

Edge case

Let’s imagine that an attacker manages to put himself in man-in-the-middle position between a client and a domain controller, and that he receives an authentication request via SMB. Knowing that a domain controller requires SMB signing, it is not possible for the attacker to relay this authentication via SMB. On the other hand, it is possible to change protocol, as we have seen above, and the attacker decides to relay to the LDAPS protocol, as authentication and session should be independant.

Well, they are almost independent.

Almost, because we saw that in the authentication data, there was the NEGOTIATE_SIGN flag which was only present to indicate whether the client and server supported signing. But in some cases, this flag is taken into account, as we saw with LDAP.

Well for LDAPS, this flag is also taken into account by the server. If a server receives an authentication request with the NEGOTIATE_SIGN flag set to 1, it will reject the authentication. This is because LDAPS is LDAP over TLS, and it is TLS layer that handles packet signing (and encryption). Thus, an LDAPS client has no reason to indicate that it can sign its packets, and if it claims to be able to do so, the server laughs at it and slams the door.

Now in our attack, the client we’re relaying wanted to authenticate via SMB, so yes, it supports packet signing, and yes, it sets the NEGOTIATE_SIGN flag to 1. But if we relay its authentication, without changing anything, via LDAPS, then the LDAPS server will see this flag, and will terminate the authentication phase, no question asked.

We could simply modify the NTLM message and remove the flag, couldn’t we? If we could, we would, it would work well. Except that there is also a signature at NTLM level.

That signature, it’s called the MIC, or Message Integrity Code.

MIC - Message Integrity Code

The MIC is a signature that is sent only in the last message of an NTLM authentication, the AUTHENTICATE message. It takes into account the 3 messages. The MIC is computed with HMAC_MD5 function, using as a key that depends on the client’s secret, called the session key.

HMAC_MD5(Session key, NEGOTIATE_MESSAGE + CHALLENGE_MESSAGE + AUTHENTICATE_MESSAGE)

What is important is that the session key depends on the client’s secret, so an attacker can’t re-compute the MIC.

Here’s an example of MIC:

Therefore, if only one of the 3 messages has been modified, the MIC will no longer be valid, since the concatenation of the 3 messages will not be the same. So you can’t change the NEGOTIATE_SIGN flag, as suggested in our example.

What if we just remove the MIC? Because yes, the MIC is optional.

Well it won’t work, because there is another flag that indicates that a MIC will be present, msAvFlags. It is also present in NTLM response and if it is 0x00000002, it tells the server that a MIC must be present. So if the server doesn’t see the MIC, it will know that there is something going on, and it will terminate the authentication. If the flag says there must be a MIC, then there must be a MIC.

All right, and if we set that msAcFlags to 0, and we remove the MIC, what happens? Since there’s no more MICs, we can’t verify that the message has been changed, can we?

…

Well, we can. It turns out that the NTLMv2 Hash, which is the response to the challenge sent by the server, is a hash that takes into account not only the challenge (obviously), but also all the flags of the response. As you may have guessed, the flag indicating the MIC presence is part of this response.

Changing or removing this flag would make the NTLMv2 hash invalid, since the data will have been modified. Here what it looks like.

The MIC protects the integrity of the 3 messages, the msAvFlags protects the presence of the MIC, and the NTLMv2 hash protects the presence of the flag. The attacker, not being aware of the user’s secret, cannot re-compute this hash.

So you will have understood that, we can do nothing in this case, and that’s because of the MIC.

Drop the MIC

A little review of a recent vulnerability found by Preempt that you will easily understand now.

It’s CVE-2019-1040 nicely named Drop the MIC. This vulnerability showed that if the MIC was just removed, even if the flag indicated its presence, the server accepted the authentication without flinching. This was obviously a bug that has since been fixed.

It has been integrated in ntlmrelayx tool via the use of the --remove-mic parameter.

Let’s take our earlier example, but this time with a domain controller that is still vulnerable. This is what it looks like in practice.

Our attack is working. Amazing.

For information, another vulnerability was discovered by the very same team, and they called it Drop The MIC 2.

Session key

Earlier we were talking about session and authentication signing, saying that to sign something you have to know the user’s secret. We said in the chapter about MIC that in reality it’s not exactly the user’s secret that is used, but a key called session key, which depends on the user’s secret.

To give you an idea, here’s how the session key is computed for NTLMv1 and NTLMv2.

# For NTLMv1
Key = MD4(NT Hash)

# For NTLMv2
NTLMv2 Hash = HMAC_MD5(NT Hash, Uppercase(Username) + UserDomain)
Key = HMAC_MD5(NTLMv2 Hash, HMAC_MD5(NTLMv2 Hash, NTLMv2 Response + Challenge))

Going into the explanations would not be very useful, but there is clearly a complexity difference from one version to the other. I repeat, do not use NTLMv1 in a production network.

With this information, we understand that the client can compute this key on his side, since he has all the information in hand to do so.

The server, on the other hand, can’t always do it alone, like a big boy. For local authentication, there is no problem since the server knows the user’s NT hash.

On the other hand, for authentication with a domain account, the server will have to ask the domain controller to compute the session key for him, and send it back. We saw in pass-the-hash article that the server sends a request to the domain controller in a NETLOGON_NETWORK_INFO structure and the domain controller responds with a NETLOGON_VALIDATION_SAM_INFO4 structure. It is in this response from the domain controller that the session key is sent, if authentication is successful.

The question then arises as to what prevents an attacker from making the same request to the domain controller as the target server. Well before CVE-2015-0005, nothing!

What we found while implementing the NETLOGON protocol [12] is the domain controller not verifying whether the authentication information being sent, was actually meant to the domain-joined machine that is requesting this operation (e.g. NetrLogonSamLogonWithFlags()). What this means is that any domain-joined machine can verify any pass-through authentication against the domain controller, and to get the base key for cryptographic operations for any session within the domain.

So obviously, Microsoft has fixed this bug. To verify that only the server the user is authenticating to has the right to ask for the session key, the domain controller will verify that the target machine in the AUTHENTICATE response is the same as the host making the NetLogon request.

In the AUTHENTICATE response, we detailed the presence of msAvFlags indicating whether or not the MIC is present, but there is also other information, such as the Netbios name of the target machine.

This is the name that is compared with the host making the NetLogon request. Thus, if the attacker tries to make a NetLogon request for the session key, since the attacker’s name does not match the targeted host name in NTLM response, the domain controller will reject the request.

Finally, in the same way as msAvFlags, we cannot change the machine name on the fly in the NTLM response, because it is taken into account in the calculation of the NTLMv2 response.

A vulnerability similar to Drop the MIC 2 has been discovered recently by Preempt security team.

Channel Binding

We’re going to talk about one last notion. Several times we have repeated that the authentication layer, thus NTLM messages, was almost independent of the application layer, the protocol in use (SMB, LDAP, …). I say “almost” because we have seen that some protocols use some NTLM messages flags to know if the session must be signed or not.

In any case, as it stands, it is quite possible for an attacker to retrieve an NTLM message from protocol A, and send it back using protocol B. This is called cross-protocol relay as we already mentioned.

Well, a new protection exists to counter this attack. It’s called channel binding, or EPA (Enhanced Protection for Authentication). The principle of this protection is to bind the authentication layer with the protocol in use, even with the TLS layer when it exists (LDAPS or HTTPS for example). The general idea being that in the last NTLM AUTHENTICATE message, a piece of information is put there and cannot be modified by an attacker. This information indicates the desired service, and potentially another information that contains the target server’s certificate’s hash.

We’ll look at these two principles in a little more detail, but don’t you worry, it’s relatively simple to understand.

Service binding

This first protection is quite simple to understand. If a client wishes to authenticate to a server to use a specific service, the information identifying the service will be added in the NTLM response.

This way, when the legitimate server receives this authentication, it can see the service that was requested by the client, and if it differs from what is actually requested, it will not agree to provide the service.

Since the service name is in the NTLM response, it is protected by the NtProofStr response, which is an HMAC_MD5 of this information, the challenge, and other information such as msAvFlags. It is computed with the client’s secret.

In the example shown in the last diagram, we see a client trying to authenticate itself via HTTP to the server. Except that the server is an attacker, and the attacker replays this authentication to a legitimate server, to access a network share (SMB).

Except that the client has indicated the service he wants to use in his NTLM response, and since the attacker cannot modify it, he has to relay it as is. The server then receives the last message, compares the service requested by the attacker (SMB) with the service specified in the NTLM message (HTTP), and refuses the connection, finding that the two services do not match.

Concretely, what is called service is in fact the SPN or Service Principal Name. I dedicated a whole article to the explanation of this notion.

Here is a screenshot of a client sending SPN in its NTLM response.

We can see that it indicates to use the CIFS service (equivalent to SMB, just a different terminology). Relaying this to an LDAP server that takes this information into account will result in a nice Access denied from the server.

But as you can see, not only there is the service name (CIFS) but there is also the target name, or IP address. It means that if an attacker relays this message to a server, the server will also check the target part, and will refuse the connexion because the IP address found in the SPN does not match his IP address.

So if this protection is supported by all clients and all servers, and is required by every server, it mitigaes all relay attempts.

TLS Binding

This time, the purpose of this protection is to link the authentication layer, i.e. NTLM messages, to the TLS layer that can potentially be used.

If the client wants to use a protocol encapsulated in TLS (HTTPS, LDAPS for example), it will establish a TLS session with the server, and it will compute the server certificate hash. This hash is called the Channel Binding Token, or CBT. Once computed, the client will put this hash in its NTLM response. The legitimate server will then receive the NTLM message at the end of the authentication, read the provided hash, and compare it with the real hash of its certificate. If it is different, it means he wasn’t the original recipient of the NTLM exchange.

Again, since this hash is in the NTLM response, it is protected by the NtProofStr response, like the SPN for Service Binding.

Thanks to this protection, the following two attacks are no longer possible :

If an attacker wishes to relay information from a client using a protocol without a TLS layer to a protocol with a TLS layer (HTTP to LDAPS, for example), the attacker will not be able to add the certificate hash from the target server into the NTLM response, since it cannot update the NtProofStr.
If an attacker wishes to relay a protocol with TLS to another protocol with TLS (HTTPS to LDAPS), when establishing the TLS session between the client and the attacker, the attacker will not be able to provide the server certificate, since it does not match the attacker’s identity. It will therefore have to provide a “homemade” certificate, identifying the attacker. The client will then hash this certificate, and when the attacker relays the NTLM response to the legitimate server, the hash in the response will not be the same as the hash of the real certificate, so the server will reject the authentication.

Here is a diagram to illustrate the 2nd case. Seems complicated, but it’s not.

It shows the establishment of two TLS sessions. One between the client and the attacker (in red) and one between the attacker and the server (in blue). The client will retrieve the attacker’s certificate, and calculate a hash, cert hash, in red.

At the end of the NTLM exchanges, this hash will be added in the NTLM response, and will be protected since it is part of the encrypted data of the NTLM response. When the server receives this hash, it will hash his own certificate, and seeing that it is not the same result, it will refuse the connection.

What can be relayed?

With all this information, you should be able to know which protocols can be relayed to which protocols. We have seen that it is impossible to relay from SMB to LDAP or LDAPS, for example. On the other hand, any client that does not set the NEGOTIATE_SIGN flag can be relayed to LDAP if the signature is not required, or LDAPS is channel binding is not required.

As there are many cases, here is a table summarizing some of them.

I think we can’t relay LDAPS or HTTPS since the attacker doesn’t have a valid certificate but let’s say the client is permissive and allowed untrusted certificates. Other protocols could be added, such as SQL or SMTP, but I must admit I didn’t read the documentation for all the protocols that exist. Shame on me.

Stop. Using. NTLMv1.

Here is a little “fun fact” that Marina Simakov suggested me to add. As we discussed, NTLMv2 hash takes into account the server’s challenge, but also the msAvFlags flag which indicates the presence of a MIC field, the field indicating the NetBios name of the target host during authentication, the SPN in case of service binding, and the certificate hash in case of TLS binding.

Well NTLMv1 protocol doesn’t do that. It only takes into account the server’s challenge. In fact, there is no longer any additional information such as the target name, the msAvFlags, the SPN or the CBT.

So, if an NTLMv1 authentication is allowed by a server, an attacker can simply remove the MIC and thus relay authentications to LDAP or LDAPS, for example.

But more importantly, he can make NetLogon requests to retrieve the session key. Indeed, the domain controller has no way to check if he has the right to do so. And since it won’t block a production network that isn’t completely up to date, it will kindly give it to the attacker, for “retro-compatibility reasons”.

Once he has the session key, he can sign any packet that he wants. It means that even if the target requests signing, he will be able to do so.

This is by design and it can not be fixed. So I repeat, do not allow NTLMv1 in a production network.

Conclusion

Well, that’s a lot of information.

We have seen here the details of NTLM relay, being aware that authentication and session that follows are two distinct notions allowing to do cross-protocol relay in many cases. Although the protocol somehow includes authentication data, it is opaque to the protocol and managed by SSPI.

We have also shown how packet signing can protect the server from man-in-the-middle attacks. To do this, the target must wait for signed packet coming from the client, otherwise the attacker will be able to pretend to be someone else without having to sign the messages he sends.

We saw that MIC was very important to protect NTLM exchanges, especially the flag indicating whether packets will be signed for certain protocols, or information about channel binding.

We ended by showing how channel binding can link the authentication layer and the session layer, either via the service name or via a link with the server certificate.

I hope this long article has given you a better understanding of what happens during an NTLM relay attack and the protections that exist.

Since this article is quite substantial, it is quite likely that some mistakes have slipped in. Feel free to contact me on twitter or on my Discord Server to discuss this.

Kerberoasting

Thu, 26 Mar 2020 08:02:44 +0000

With the help of previously discussed notions, we have everything in hand to explain the Kerberoasting attack principle, based on the TGS request and the SPN attributes of Active Directory accounts.

Principle

The article on how kerberos works helped to understand how a user requests a TGS from the domain controller. The KRB_TGS_REP response is composed of two parts.

The first part is the TGS whose content is encrypted with the secret of the requested service
The second part is a session key which will be used between the user and the service. The whole is encrypted using the user’s secret.

A domain user can ask a domain controller for a TGS for any service. It is not the role of the domain controller to check access rights. The only purpose of the domain controller when asked for a TGS is to provide security information related to a user (via the PAC). It is the service who must check the user’s rights by reading the PAC provided in the TGS.

For example, TGS request can be made by specifying arbitrary SPN. If those SPN are registered in the Active Directory, the domain controller will provide a piece of information encrypted with the secret key of the account executing the service. With this information, the attacker can now try to recover the account’s plaintext password via a brute-force attack.

Fortunately, most of the accounts that runs services are machine accounts (MACHINENAME$) and their password are very long, complex and completely random, so they’re not really vulnerable to this type of attack. However, there are some services executed by accounts whose password have been chosen by a humans. It is those accounts that are much simpler to compromise via brute-force attack, so it is those accounts which will be targeted by a Kerberoast attack.

In order to list those accounts, a LDAP filter can be used to extract user-type accounts with a non-empty servicePrincipalName. The filter is as follow:

&(objectCategory=person)(objectClass=user)(servicePrincipalName=*)

Here is a simple PowerShell script allowing you to retrieve users with at least one SPN :

$search = New-Object DirectoryServices.DirectorySearcher([ADSI]"")
$search.filter = "(&(objectCategory=person)(objectClass=user)(servicePrincipalName=*))"
$results = $search.Findall()
foreach($result in $results)
{
	$userEntry = $result.GetDirectoryEntry()
	Write-host "User : " $userEntry.name "(" $userEntry.distinguishedName ")"
	Write-host "SPNs"        
	foreach($SPN in $userEntry.servicePrincipalName)
	{
		$SPN       
	}
	Write-host ""
}

In my lab, a fake SPN as been set on user support-account.

Thus, during our LDAP search, here is what we get:

Of course, there are several tools to automate this task. For instance Invoke-Kerberoast.ps1 by @Harmj0y, which takes care of listing user accounts with one or more SPN, request some TGS for those accounts and extract the encrypted part in a format that can be cracked (by John for example).

Invoke-Kerberoast -domain adsec.local | Export-CSV -NoTypeInformation output.csv
john --session=Kerberoasting output.csv

Or GetUserSPNs.py provided by Impacket.

We then hope to find password, which depends on the company’s password policy for these accounts.

Protection

To protect ourselves against this attack, we must avoid having SPN on user accounts, in favor of machine accounts.

If it really is necessary, then we should use Microsoft’s Managed Service Accounts (MSA) feature , which ensures that the account password is robust and changed regularly and automatically. To do so, simply add a service account (only via PowerShell):

New-ADServiceAccount sql-service

Then it has to be installed on the machine :

Install-ADServiceAccount sql-service

Finally, this user must be assigned to the service.

Conclusion

The Kerberoast attack allows us to retrieve new accounts within an Active Directory for lateral movement. The compromised accounts can have higher privileges, which is sometimes the case on the computer hosting the service. It is then important from a defensive perspective to control the SPN attribute on domain accounts to prevent accounts with weak password from being vulnerable to this attack.

AS_REP Roasting

Thu, 19 Mar 2020 07:10:06 +0000

When asking for a TGT, by default, a user has to authenticate himself to the domain controller in order to get a response. Sometimes, no authentication is asked before returning a TGT for specific account, allowing an attacker to abuse this configuration.

Preamble

When we talk about TGT it’s often a language abuse, because we are talking about the KRB_AS_REP which contains the TGT (encrypted by the domain controller’s secret) and the session key (encrypted with the user account secret).

In this article, the TGT notion will refer to the TGT contained in the KRB_AS_REP response.

Pre-authentication

When we talked about how Kerberos works, it was highlighted that during the first exchange (KRB_AS_REQ - KRB_AS_REP), the client must first authenticate himself to the domain controller, before obtaining a TGT. A part of the response of the domain controller being encrypted with the client’s account secret (the session key), it is important that this information is not accessible without authentication. Otherwise, anyone could ask for a TGT for a given account, and try to decrypt the encrypted part of the response KRB_AS_REP in a brute-force way in order to recover the password of the targeted user.

That’s why the user, in his KRB_AS_REQ request, must send an authenticator encrypted with his own secret in order for the domain controller to decrypt it and send back the KRB_AS_REP if it is successful. If an attacker asks for a TGT with an account he does not have control over, he won’t be able to encrypt the authenticator correctly, therefore the domain controller will not return the desired information.

This is the default behavior, it protects the accounts against this offline attack.

KRB_AS_REP Roasting

However, for some strange reason (dark one though), it is possible to disable the pre-authentication prerequisite for one or more account(s).

For example in this article, the author states that in order to benefit from SSO on a database hosted on a Unix server, he has to disable the pre-authentication for the user. It remains a very rare case, and even cptjesus and Harmj0y don’t really have an answer.

cptjesus > As far as why its disabled, I couldn’t tell you

Harmj0y > I honestly don’t really know why it would be disabled, just have heard from a people about the linux/”old” angle.

Anyway, if this option is disabled, anyone could ask for a TGT in the name of one of these accounts, without sending any authenticator, and the domain controller will send back a KRB_AS_REP.

This can be done with the ASREPRoast tool of @Harmj0y or more recently with Rubeus using asreproast functionnality.

There is also impacket GetNPUsers.py tool that can perform this operation.

Once in possession of the domain controller response KRB_AS_REP, the attacker can try to find out the victim’s clear text password offline, by using John The Ripper with the krb5asrep mode, or with hashcat for example.

Conclusion

This technique, also described in an article wrote by Harmj0y, is a way to retrieve a clear text password within an Active Directory environment when you don’t have any foothold. But if you don’t have any account yet, it can be difficult to find out this information as you are not able to talk with the domain controller. An OSINT phase can be useful to enumerate as many valid account as possible, and try this attack on every account you found.

If any account is set up so that it does not need a pre-authentication, an attacker could simply ask for a TGT for this account and try to recover its password. With powerful machine, the cracking speed can be really huge. However, you should be aware that accounts without the necessary pre-authentication required are pretty rare. They can exist for historical reason, but kerberoasting is still more widespread.